An AI agent that only chats can say something wrong. An agent that reads your inbox, queries your CRM and can send email can do something wrong, and that is a different class of problem. Prompt injection is the reason: text the agent reads can change what the agent does.
Most teams meet this the first time a pilot touches real data. Someone asks whether we can "just add a filter" for malicious prompts. The honest answer, from the sources below, is that a filter helps at the margin and cannot be the thing you rely on. This article covers what the security guidance says, then a design approach we would use on a client build: limit what a manipulated agent can reach, enforce that limit in ordinary code, and keep a human in the loop where the cost of a mistake is high.
Everything here comes from published guidance and incident write-ups, cited at the end. We have not run our own attack benchmarks, so we make no claims about how often attacks succeed.
What prompt injection is, and why it is hard
OWASP's LLM01:2025 entry defines a prompt injection vulnerability as one where user prompts alter the model's behaviour or output in unintended ways. It separates two forms. In direct injection, the person typing the prompt changes the behaviour. In indirect injection, the model accepts content from an external source such as a website or a file, and that content changes behaviour when the model interprets it. For agents, indirect injection is the one to worry about, because the whole point of an agent is that it reads things you did not write.
The UK National Cyber Security Centre explained in December 2025 why this is not the same as SQL injection. A database separates instructions from data, so parameterised queries can fix SQL injection properly. An LLM has no equivalent boundary, because it works by predicting the next token over everything in its context. The NCSC's conclusion is that prompt injection may never be fully mitigated in the way SQL injection can be, and that the realistic goal is to reduce both its likelihood and its impact. It describes the model as an "inherently confusable deputy", and it advises designing on the assumption that the model can be manipulated.
If you take one thing from this article, take that framing. You are not trying to build a model that never gets fooled. You are trying to build a system in which a fooled model has little to work with.
A real example of the pattern
EchoLeak (CVE-2025-32711) is a useful case study because it shows the mechanics without needing any exotic technique. Aim Security reported a critical flaw in Microsoft 365 Copilot, scored 9.3 on CVSS. An attacker sent an ordinary-looking email containing hidden instructions. Later, when a user asked Copilot a business question, its retrieval step pulled that email into the same context as sensitive internal data. The injected instructions then caused the data to leak out through Microsoft-hosted URLs, with no click from the user. Microsoft fixed it server-side as part of its June 2025 Patch Tuesday, and press coverage at the time reported no evidence of exploitation in the wild.
Look at what had to be true for that to work: the assistant read outside content, it could reach private data, and it had a path to send information out. Those three conditions are the pattern to design against.
The three properties to watch
Simon Willison named a version of this in June 2025 as the "lethal trifecta": access to your private data, exposure to untrusted content, and the ability to communicate externally. If an agent has all three, an attacker who can get text in front of it may be able to get your data out. His advice for people combining tools is to avoid the full combination.
Meta AI published a closely related framework on 31 October 2025, the Agents Rule of Two. It uses three properties:
- A: the agent can process untrustworthy inputs.
- B: the agent can access sensitive systems or private data.
- C: the agent can change state or communicate externally.
Meta's guidance is that, until research can reliably detect and refuse prompt injection, an agent should satisfy no more than two of the three within a session. If a task genuinely needs all three, the agent should not run autonomously for that session and needs supervision, such as human approval or another reliable check. Meta also notes that the properties apply per session, so a fresh context can change the picture.
Meta is clear about the limits. The rule does not address mistakes, hallucinations or excessive privileges, and a compliant design can still fail if a user approves a warning without reading it. We treat it as a design review question, not a certificate.
Using it in a design review
Before building an agent, list its inputs, data access and actions, and mark each as A, B or C. The result often changes the architecture. A few patterns we would consider:
- A support-triage agent reads inbound tickets (A) and customer records (B). Let it draft a reply and a suggested category, but do not give it a send-email tool (C). A person sends the reply. This is an A+B design.
- A research agent browses the public web (A) and writes notes (C) in a store that holds no private data. No B, so there is little to steal.
- A workflow that needs all three, such as an agent that reads customer email, looks up account data and replies on its own, should split into stages. A first stage reads and extracts structured fields from the untrusted text. A second stage, which never sees the raw text, acts on those fields. This reduces exposure but does not remove it, because extraction can be manipulated too, so the second stage still needs the gate described below.
Treat the model's output as a proposal
The most useful rule we know is this: the model proposes, deterministic code decides. OWASP's mitigation list points the same way, including privilege control with least-privilege access, human approval for high-risk actions, and validating expected output formats. The NCSC makes the same point, advising non-LLM safeguards that constrain what the model can do with tools and APIs, so that processed input from an untrusted party does not grant privileges.
Here is a minimal sketch of a policy gate in TypeScript. It is illustrative, not a library. The tool names, limits and the customerId check are examples for you to replace with your own rules.
type ToolCall = { name: string; args: Record<string, unknown> };
type Decision =
| { action: "allow" }
| { action: "needs_approval"; reason: string }
| { action: "deny"; reason: string };
// Everything the current task may touch, decided by your app before the
// model runs, never by the model.
interface TaskScope {
allowedTools: ReadonlySet<string>;
customerId: string;
allowedEmailDomains: ReadonlySet<string>;
}
export function checkToolCall(call: ToolCall, scope: TaskScope): Decision {
if (!scope.allowedTools.has(call.name)) {
return { action: "deny", reason: `tool ${call.name} not in scope` };
}
if (call.name === "lookup_order") {
// The model may supply an order id, but never widen the customer.
if (call.args.customerId !== scope.customerId) {
return { action: "deny", reason: "cross-customer access" };
}
return { action: "allow" };
}
if (call.name === "send_email") {
const to = String(call.args.to ?? "");
const domain = to.split("@")[1]?.toLowerCase() ?? "";
if (!scope.allowedEmailDomains.has(domain)) {
return { action: "deny", reason: "recipient domain not allowed" };
}
return { action: "needs_approval", reason: "outbound email" };
}
// Default deny: a new tool is blocked until someone writes a rule for it.
return { action: "deny", reason: "no policy for this tool" };
}Three design choices matter more than the code. The scope object is built from the authenticated user and the task, before the model sees anything. Anything not explicitly handled is denied. And the sensitive checks, such as which customer's data a call may touch, use values the application controls, not values the model typed.
Log every decision, including denials. The NCSC recommends logging inputs, outputs and tool activity, and notes that failed tool calls can be a sign of an attack in progress. A burst of denied send_email calls after the agent read a new document is exactly the signal you want to be alerted on.
Controls that help, and controls that don't carry the weight
It is worth being direct about which controls are load-bearing.
Marking untrusted content is worth doing. The NCSC suggests separating data from instructions with techniques such as XML-style tags, and OWASP lists segregating and identifying external content. Both are described as making attacks harder, not as solving the problem. Put external text in a clearly delimited block, say in the system prompt that its contents are data, and keep it out of the instruction section.
Input and output filtering is also on OWASP's list. Use it for what it is good at, such as blocking known-bad output patterns or stripping markup that could render as an outbound request. The NCSC advises against relying on deny-lists of known bad phrases and cautions against products that claim to stop prompt injection outright. A filter is a speed bump.
Human approval is on OWASP's list for high-risk actions, and it works if the approver sees what will actually happen. Show the exact recipient, amount or record and the literal arguments, not the model's description of them. If every action needs approval, people stop reading, so reserve it for the actions that matter and let low-risk, reversible ones through.
Least privilege is the control with the most leverage. Issue each task a narrow, short-lived credential. A read-only database role for a summarising agent means a successful injection cannot write. Network egress limits mean it cannot reach arbitrary hosts. This is ordinary security engineering, which is the point.
Adversarial testing is OWASP's seventh item. Build a small set of hostile documents, emails and web pages for your own use case and run them against the agent whenever you change a prompt, a model or a tool. Include cases where the injected text asks for data the agent legitimately holds and tries to move it out through each channel you expose.
Where MCP and tool servers fit
Connecting an agent to external tool servers widens the A and C surfaces. Every new server adds tool descriptions and tool results that the model reads, and results from third-party systems count as untrusted input. Apply the same discipline: allow-list the tools per task, treat tool output as data, and run the policy gate on every call regardless of which server it came from. If you are planning a Model Context Protocol rollout, our MCP 2026-07-28 migration guide covers the protocol side, and tracing AI agents with OpenTelemetry covers capturing the tool-call logs this approach depends on.
A pre-launch checklist
- For each agent, write down inputs, data access and actions, and mark A, B and C. If all three are present, redesign or require approval.
- Build the task scope from the authenticated user, not from model output.
- Put a deny-by-default policy gate between the model and every tool.
- Issue narrow, short-lived credentials per task. Prefer read-only roles where writes are not needed.
- Restrict outbound network access and strip or block markup that triggers automatic requests.
- Require approval for irreversible or external actions, showing the literal arguments.
- Log inputs, outputs, tool calls and denials, and alert on unusual denial patterns.
- Keep a hostile test set and re-run it on every prompt, model or tool change.
- Record the residual risk and who accepted it. The NCSC advises that risk owners acknowledge a remaining risk exists, and that if a system cannot tolerate it, an LLM may not be the right choice.
What this means for your project
None of this is exotic. It is access control, input handling and review, applied to a component that behaves less predictably than most. The teams that get into trouble tend to give an agent broad access first and plan to add safeguards later. It is much cheaper to decide the A, B and C tradeoffs before the first tool is wired up.
If you are building an agent that touches customer data and want a second review of its permissions and approval flow, our team does this as part of AI automation work, and we can review an existing design through consulting. If the agent is part of a larger product, see custom software development. You can also book a call to talk through your case.
Sources
- OWASP GenAI Security Project, "LLM01:2025 Prompt Injection" (genai.owasp.org/llmrisk/llm01-prompt-injection): definitions of direct and indirect injection and the list of seven mitigations.
- UK National Cyber Security Centre, "Prompt injection is not SQL injection (it may be worse)", Dave Chismon, 8 December 2025 (ncsc.gov.uk): why the analogy fails, residual risk, and recommended design and monitoring measures.
- Meta AI, "Agents Rule of Two: A Practical Approach to AI Agent Security", 31 October 2025 (ai.meta.com/blog/practical-ai-agent-security).
- Simon Willison, "The lethal trifecta for AI agents", 16 June 2025 (simonwillison.net/2025/Jun/16/the-lethal-trifecta).
- The Hacker News, "Zero-Click AI Vulnerability Exposes Microsoft 365 Copilot Data Without User Interaction", June 2025 (thehackernews.com): EchoLeak, CVE-2025-32711, CVSS 9.3, reported by Aim Security and fixed by Microsoft.
