Navigating the AI WorldPart C · Building AI into workChapter 19 of 28

Design for hostile and unreliable input

A poisoned document becomes much more serious once the assistant has somewhere useful to send the result. When a system can read files, call business tools or reach the network, prompt injection is an authority problem as well as a model-behaviour problem. Direct injection comes from the user's input. Indirect injection arrives through documents, web pages, tickets, tool results or other material the system reads. [3]

The design has to remain safe when the model makes the wrong decision. Validate tool arguments, restrict network destinations, keep credentials outside context and enforce permissions in the services that release data or perform actions. Whenever generated text will be consumed by another interpreter, query engine or user interface, treat it as untrusted input.

Failure or attackConcrete control to test
A retrieved page asks for a secret uploadDestination restrictions and narrow tools block the transfer
A generated query reaches another tenantServer-side tenant scoping and record authorization deny access
Model output contains active HTML or commandsAppropriate encoding, validation and isolated execution prevent interpretation
A poisoned document changes a policy answerSource ownership, version review and evidence checks expose the change
Repeated calls exhaust resourcesPer-user quotas, step limits, timeouts and concurrency limits stop the run
A changed package or model introduces riskReviewed provenance, pinned artifacts and staged updates constrain deployment

Keep credentials and business secrets in controlled stores, not in the system prompt. Assume prompt content may become visible through debugging, product behaviour or attack techniques. Put only the information needed for the authorized task into model-visible context.

Supply-chain review extends beyond application libraries. It includes model files, tokenizers, loaders, datasets, containers and integration servers. Some loading paths can execute code. An open-weight download tells you little by itself about provenance, licensing or runtime safety. Review the artifacts you will actually execute and the permissions they receive.

Authorization is only one part of AI risk. Bad advice, biased treatment, privacy loss and overreliance can harm people without an attacker being present. Keep the threat model connected to the governance work in chapter 20 and the professional judgement risks introduced in chapter 4.

The controls here are a practical synthesis informed by the OWASP LLM and agentic guidance. They are not a replacement for the full sources. Use the referenced OWASP material when reviewing a real deployment because the risk set is broader than one chapter can cover. [3–4]

Report a correction

Corrections go to the editor and are never published automatically. No account needed.