Start Here: Prompt Injection Is Not Solved, and Nothing Below Makes Your Agent Safe
Every claim in this article sits under one constraint, so read it first. Prompt injection is not a solved problem, and no control described here — not classifiers, not delimiters, not adversarial training, not capability-based information-flow control — makes an agent with tool access safe. The correct frame throughout is defense-in-depth and blast-radius reduction: assume the model's instruction boundary will be bypassed, and constrain what that bypass is able to reach.
That is not our framing. It is the framing of the organizations that build and defend these systems. The OWASP GenAI LLM Top 10, Version 2026 opens with it. OpenAI writes that prompt injection “much like scams and social engineering on the web, is unlikely to ever be fully ‘solved’” and that it “remains an open challenge for agent security, and one we expect to continue working on for years to come.” Google DeepMind writesthat “even with model hardening, no model is completely immune,” and that indirect prompt injection “is not the kind of technical problem you ‘solve’ and move on.”
Stop trying to build a model that cannot be fooled. Build the system around it, so that when the model is fooled, and it will be, nothing important breaks.
— OWASP Top 10 for LLM Applications, Version 2026 — Letter from the Project Leads
The rest of this piece is the engineering consequence of that sentence. It covers what the two current OWASP lists actually say and how they changed, why the problem is structural rather than an implementation gap, the adaptive-attack evidence that invalidated most published defenses, the small set of architectural approaches that hold up and what they cost, the two structural checks worth running before any code exists, the incidents that show what a real exploit looks like, and a pre-production checklist.
The OWASP GenAI LLM Top 10, Version 2026 — and the Methodology That Changed
Get the versions right, because both current OWASP lists are new and both are labeled “Version 2026.” The OWASP Top 10 for LLM Applications, Version 2026 is dated August 4, 2026. It is a different document from the OWASP Top 10 for Agentic Applications, Version 2026, published in December 2025. Citing one when you mean the other is the most common error in secondary writing on this topic.
The LLM list changed methodology for the first time in its history. Previous versions were built purely on practitioner judgment. For 2026, the project pulled together a corpus of 7,714 real incidents from public vulnerability databases and an AI-harm database, built classifiers that read them, and placed the 6,639 that carried enough detail to sort. The final ranking weights the community vote at three-quarters and the incident data at one quarter— deliberately, so that, in the project's words, one noisy year of data does not overturn the judgment of the people doing the work.
The most interesting result of that exercise concerns prompt injection itself, and it is worth understanding before you read the ranking. Practitioners rank prompt injection the number one risk. Rank the categories by the raw incident record instead, and it falls out of the top ten entirely. OWASP calls that gap a defense effect: teams fight injection hard, so fewer clean exploits reach a public database, and the public count understates the risk that mature teams already spend real money holding off. Low recorded incident volume in this category measures defensive spend, not low risk.
| 2026 entry | 2025 rank | Move | What changed |
|---|---|---|---|
| LLM01:2026 — Prompt Injection | 1 | Held | Now explicitly covers cross-modal attacks that hide instructions inside an image or audio track |
| LLM02:2026 — Sensitive Information Disclosure | 2 | Held | The one place at the top where the vote and the incident record agree; OWASP states its confidence is highest here |
| LLM03:2026 — Excessive Agency | 6 | ▲ 3 places | OWASP calls this the most consequential move on the list — vote and record agree that agentic deployments are where damage lands |
| LLM04:2026 — Supply Chain | 3 | ▼ 1 place | Extended to cover the trust failure when a promoted model artifact is not what it claims to be |
| LLM05:2026 — Data and Model Poisoning | 4 | ▼ 1 place | Now absorbs fine-tuning subversion |
| LLM06:2026 — Unbounded Consumption | 10 | ▲ 4 places | Carried by practitioners weighting resource and cost exhaustion higher than its old rank |
| LLM07:2026 — Misinformation | 9 | ▲ 2 places | The widest belief-versus-evidence gap: voters placed it near the bottom, the incident record near the top |
| LLM08:2026 — Hidden Context Exposure | 7 (System Prompt Leakage) | Renamed and re-scoped | A broader frame for the same failure to trust information that should have stayed out of reach |
| LLM09:2026 — Vector and Embedding Weaknesses | 8 | ▼ 1 place | Retrieval and embedding stores treated as a first-class disclosure surface |
| LLM10:2026 — Improper Output Handling | 5 | ▼ 5 places | The furthest fall on the list; now spans the insecure code assistants generate at scale |
OWASP Top 10 for LLM Applications, Version 2026 (August 4, 2026), with movement against the 2025 list.
Four moves matter for anyone building agents. Excessive Agency climbed from sixth to third — OWASP calls it the most consequential move on the list, because the vote and the incident record agree that agentic deployments are where the damage is landing. Unbounded Consumption rose four places, from tenth to sixth. Improper Output Handling fell the furthest, from fifth to tenth, though it now also spans the insecure code that assistants generate at scale. And System Prompt Leakage was renamed and expanded into Hidden Context Exposure, a broader frame for the same failure to trust information that should have stayed out of reach.
Read the Misinformation entry even though it is seventh
Misinformation is the widest belief-versus-evidence gap on the list, and it runs in the dangerous direction: voters placed it near the bottom, the incident record placed it near the top. The list seats it in the middle because the vote carries greater weight, but the disagreement is the point.
The mechanism is directly relevant to agent engineering. When a model's fluent, confident output drives a decision or a tool call, a wrong answer becomes a wrong action. In an agent, the boundary between “it said something incorrect” and “it did something incorrect” is one function call wide.
The Separate Agentic Top 10, and Where the Scope Boundary Sits
The OWASP Top 10 for Agentic Applications, Version 2026 came out of the project's Agentic Security Initiative in December 2025. Its entries are numbered ASI01 through ASI10, and they describe failures that only exist once a model plans, holds memory, calls tools, and acts with delegated authority.
| Entry | What it covers |
|---|---|
| ASI01 — Agent Goal Hijack | Attacker manipulation of objectives, task selection, or decision pathways — via prompts, deceptive tool outputs, malicious artefacts, forged agent-to-agent messages, or poisoned external data |
| ASI02 — Tool Misuse and Exploitation | Unsafe tool actions steered by external content, plus tool poisoning that corrupts the tool interface itself, including MCP tool descriptors |
| ASI03 — Identity and Privilege Abuse | The agent acting under credentials and scopes far broader than the task requires |
| ASI04 — Agentic Supply Chain Vulnerabilities | Compromise arriving through MCP servers, agent cards, agentic registries, and third-party tool packages |
| ASI05 — Unexpected Code Execution (RCE) | Generated or induced code reaching an interpreter, shell, or build system |
| ASI06 — Memory & Context Poisoning | Persistent corruption of stored context or long-term memory that outlives the session that planted it |
| ASI07 — Insecure Inter-Agent Communication | Agent-in-the-middle attacks, spoofed descriptors, protocol downgrade across MCP, A2A, and gRPC |
| ASI08 — Cascading Failures | One compromised or wrong step propagating through a multi-agent chain |
| ASI09 — Human-Agent Trust Exploitation | Attacks on the approval surface and on the human's willingness to believe what the agent displays |
| ASI10 — Rogue Agents | Autonomous misalignment that emerges without an active attacker in the loop |
OWASP Top 10 for Agentic Applications, Version 2026 (December 2025) — ASI01 through ASI10.
The scope boundary between the two lists is stated explicitly in the 2026 LLM document, and it is the cleanest available answer to “which list applies to me.” The LLM list owns the risk when the model is a component inside your application. The moment that model becomes an actor — with tools it can call, memory it carries between sessions, and consequences it sets in motion downstream — the risk moves to the Agentic list. OWASP notes that many of the incidents it read sit right on that boundary, and instructs readers to pair the entries, because neither list covers that ground alone.
Why It Is Structurally Unsolved: No Parameterized Queries, Three Amplifiers
SQL injection had a fix. Parameterized queries work because the database engine receives the query structure and the data through separate channels, and the structure is fixed before user data arrives. There is no equivalent for a language model, and OWASP states the reason in one sentence.
LLMs make no architectural distinction between “instructions” and “data” (both are tokens on the same stream), so there is no clean equivalent to parameterized queries.
— OWASP Top 10 for LLM Applications, Version 2026 — LLM01:2026 Prompt Injection
That is the whole problem, and it explains why every in-band defense — every delimiter, marker, warning, hierarchy, or classifier — is a heuristic operating inside the same channel it is trying to police. It also explains why the definition is broader than most engineers assume: inputs need not be human-readable, need not arrive directly from a user, and need not be visible in the rendered interface to change model behavior.
On top of that base problem, OWASP names three deployment-time properties that turn a model-level weakness into a system-level breach. These are the three things to look for in your own architecture diagram.
- Context-window pooling: The model treats system prompt, user input, retrieved documents, tool outputs, conversation history, and memory as a single token stream, with no enforced trust boundary between them. Everything that lands in the window has the same standing as everything else.
- Memory persistence: An injection that writes to long-term memory, a RAG corpus, a vector store, or a hosted memory service taints every subsequent session that reads from that store. One successful write becomes a permanent compromise until someone finds and removes it — and nobody is looking, because it does not present as an incident.
- Agentic execution: When model output drives tool calls — file system, shell, email, cloud APIs, MCP servers, sub-agents — the blast radius extends from the chat surface to whatever those tools can reach. Tool outputs then re-enter the context window, which is what enables chained and self-replicating effects.
The third one deserves the emphasis it gets in the 2026 list. OWASP's own reading of the incident corpus is that most high-impact prompt-injection incidents on record became severe not because the injection was clever, but because it landed inside a system whose tools, scopes, or output-rendering capabilities let the compromised model act on the attacker's behalf at the user's privilege level. Severity is a property of your architecture, not of the payload.
There is also a taxonomy worth adopting for threat modeling, because it turns a vague fear into three answerable questions. OWASP characterizes any injection along three axes: delivery surface (direct input, retrieved content, tool output, tool connection channel, or persistent memory), propagation behavior (single-shot, multi-step kill chain, cross-session through memory or RAG, or self-replicating across agents), and encoding (plain text, base64 or other obfuscation, invisible Unicode, multimodal or steganographic, low-resource language). Decompose your scenario on those three axes before choosing mitigations, or you will buy a text filter for an image payload.
The trust-profile distinction that changes what defenses are practical
OWASP splits indirect injection surfaces three ways, and the split matters because it determines what you can realistically do. Untrusted surfaces — public web pages, emails from unknown senders, search results — are where most research has focused and where suspicion is cheap. Semi-trusted surfaces — issue titles in a public bug tracker, package READMEs and changelogs, third-party API responses — are content the user chose to retrieve but did not author. Trusted surfacesare the developer's own repositories, databases, internal documents, and mail.
The pattern that keeps producing real incidents targets the third category through the second: an attacker submits text into a trusted-by-the-user location through a low-privilege channel — a public form, a customer ticket, a community pull request — and waits for the user's agent to read it while operating under the user's elevated credentials. The agent, not the attacker, performs the privileged action. A defense that only inspects the chat surface misses this entirely.
The Attacker Moves Second: Twelve Defenses, One Result
This is the most important section in the article, and if you take one citation away from it, take this one. In October 2025 a joint team from OpenAI, Anthropic, Google DeepMind, ETH Zurich and Northeastern University published “The Attacker Moves Second” (arXiv:2510.09023). The argument is methodological: defenses are typically evaluated against a static set of harmful attack strings, or against weak optimization methods that were not designed with the defense in mind. Evaluate them instead against an attacker who has read the defense and spends real compute against it, and the picture inverts.
Using gradient descent, reinforcement learning, random search, and human-guided exploration, the team bypassed twelve recent defenses with attack success above 90% for most of them. The sentence that should end most vendor conversations is the researchers' own observation that the majority of those defenses had originally reported near-zero attack success rates.
| Defense | Originally reported ASR | ASR under adaptive attack | Source |
|---|---|---|---|
| Spotlighting (delimiting / marking untrusted content) | 1% | >95% | Nasr et al., arXiv:2510.09023 |
| Prompt Sandwiching | 1% | >95% | Nasr et al., arXiv:2510.09023 |
| Circuit Breakers | Reported near zero | 100% | Nasr et al., arXiv:2510.09023 |
| StruQ (structured query separation) | Reported near zero | 100% | Nasr et al., arXiv:2510.09023 |
| MetaSecAlign | 2% | 96% | Nasr et al., arXiv:2510.09023 |
| Protect AI Detector / PromptGuard / Model Armor | Vendor-reported detection | >90% | Nasr et al., arXiv:2510.09023 |
| MELON | 0.24% (ICML 2025 paper) | 76% blind, 95% informed | arXiv:2510.09023 vs arXiv:2502.05174 |
| Instruction Hierarchy (training-time) | OpenAI reported large gains, 2024 | 98% vs GPT-4o, 72% vs GPT-5 | RL-Hammer, arXiv:2510.04885 |
Static versus adaptive attack success rates. ASR = attack success rate; lower is better. Figures as published, as of August 2026.
One finding in that paper deserves separate emphasis because it changes how you should staff a red team: human red-teaming succeeded on 100% of scenarios where automated static attacks succeeded on 0%. A defense can be perfectly robust against every attack string in your test suite and fall to the first person who reads your design document.
This is not one anomalous paper. Zhan et al. (NAACL 2025 Findings) evaluated eight defenses against indirect prompt injection and bypassed all eight, with attack success consistently above 50% — one defense moving from a 9% baseline to above 50% once the attacker adapted. The pattern is stable across research groups, model families, and defense categories.
OWASP has now folded this into its own mitigation guidance for LLM01, which is a meaningful signal about where consensus sits. Control #11 in the 2026 list instructs readers to test against adaptive attackers who have read the deployed defense, and to reject static-only attack-success claims — baselining with AgentDojo and JailbreakBench, then red-teaming with the full defense specification disclosed to the testers.
Google's Numbers, and the Methodological Warning to Quote at Your Next Review
Google DeepMind's layered-defense paper (arXiv:2505.14534, May 2025) is the single most useful public dataset on this question, because it publishes the same defenses measured twice: once against non-adaptive attacks and once against adaptive ones. Undefended Gemini 2.0 was already at 94.8–100% attack success for the strongest attack family, with attack triggers costing under $10 to generate.
| Defense (Gemini 2.0) | Actor-Critic: non-adaptive → adaptive | Beam Search: non-adaptive → adaptive | TAP: non-adaptive → adaptive |
|---|---|---|---|
| In-context learning (ICL) | 46.2 → 59.8 | 34.4 → 66.6 | 67.6 → 98.4 |
| Spotlighting | 18.0 → 82.4 | 13.0 → 64.8 | 75.2 → 82.2 |
| Paraphrase | 53.0 → 31.2 | 17.6 → 4.8 | 77.4 → 95.2 |
| Warning | 5.8 → 8.4 | 0.2 → 0.0 | 1.2 → 10.8 |
| Perplexity filter | 65.4 → 75.0 | 97.8 → 68.0 | 99.8 → 62.4 |
| Self-reflection | 44.2 → 54.6 | 21.2 → 66.2 | 18.0 → 26.0 |
Attack success rate (%), non-adaptive versus adaptive, per defense and attack family. Google DeepMind, arXiv:2505.14534, May 2025 — still at version 1 as of August 2026.
Read the Spotlighting row against the vendor claim. Microsoft's original Spotlighting paper reported attack success falling from above 50% to below 2%. Here it sits at 18.0% non-adaptive and 82.4% adaptive for one attack family. Read the Paraphrase row too: it genuinely helps against two attack families and collapses to 95.2% against the third. Defenses are attack-family-specific, which is exactly why single-number claims mislead.
Then read the paragraph that should be printed and pinned above any team's benchmark dashboard. On a 1,799-prompt non-adaptive evaluation set, attack success fell from 92% on Gemini 2.0 to 18% on Gemini 2.5 — a result that looks like a decisive generational improvement in robustness. The authors ran adaptive attacks anyway.
Had we not conducted further adaptive attacks… we would have incorrectly concluded that Gemini 2.5 exhibits a higher degree of robustness than it does.
— Google DeepMind, arXiv:2505.14534
Adversarial fine-tuning from Gemini 2.0 to 2.5 produced roughly a 47% average reduction in attack success. Two data points from inside that average show why the average is misleading: on one task family, Beam Search attack success went from 98.6% to 0, while TAP went from 100% to 94.6% — barely dented. Layering helped materially: a warning defense on top of the adversarially trained Gemini 2.5 brought attack success to 6.2% where 2.5 alone had been at 94.6%. That is a real result and it is also not zero.
Independent government evaluation reaches the same place. The US AI Safety Institute found that Claude 3.5 Sonnet's agent-hijacking rate moved from 11% on baseline attacks to 81% on novel model-specific attacks — the same model, the same task, a different attacker.
On benchmark hygiene: AgentDojo, and what not to cite
AgentDojo (97 tasks, 629 security test cases) is the standard baseline and worth using. Two cautions. First, its public results page was last updated in February 2025and contains no 2026 models; its authors state explicitly that it is not a leaderboard. Any “current state of the art on AgentDojo” claim must cite an individual paper, not that page.
Second, calibrate on the baseline before reading any defense number: current models solve under 66% of AgentDojo tasks with no attack present at all. And 2026 attack work keeps moving — PIMiner (arXiv:2608.05108, August 2026) reports AgentDojo attack success of 86.7% against Gemini 2.5 Pro, 53.3% against GPT-5.1, and 40.0% against Claude Sonnet 4.5 at roughly ten queries per sample.
What Actually Holds Up: Out-of-Band Defenses, and What They Cost
The consistent finding across the adaptive-attack literature is that defenses which try to detect or resist injection inside the model's channel degrade badly, and defenses which enforce constraints outside it hold up far better. Not perfectly. Far better. OWASP frames the same conclusion as a design rule: defense is architectural rather than interceptive.
The most rigorous published result is CaMeL (Google DeepMind, arXiv:2503.18813). It works by having a privileged model emit a program in a restricted language, with capability metadata attached to every value, and a deterministic interpreter enforcing an information-flow policy at execution time. The model never holds the authority; the interpreter does.
| Approach | Reference | Reported result | The honest cost / caveat |
|---|---|---|---|
| CaMeL — capability-based information-flow control | arXiv:2503.18813 | 77% of AgentDojo tasks solved with provable security vs 84% undefended; 0 successful attacks in 949 runs (next-best defense: 8); Gemini 2.5 Pro 300 successful attacks → 0 | 2.82× input tokens, 2.73× output tokens; per-suite utility 63.92% vs 90.72% undefended, travel suite 25.00%; confirmations on 33.87% (normal) to 53.23% (strict) of prompts; self-reported side-channel vulnerability |
| IPIGuard — tool dependency graph | arXiv:2508.15310, EMNLP 2025 | AgentDojo with GPT-4o-mini: 0.69% ASR at 58.77 utility vs no-defense 13.16% / 54.30 | Not evaluated against an adaptive attacker in the published result |
| SIEVE | arXiv:2512.06716 | AgentDojo 0.34% ASR at 87.63% clean utility | Same caveat — a static number until an adaptive team reads the design |
| Progent — deterministic out-of-band policy enforcement | arXiv:2606.26479 | 25.8% → 4.2% ASR; a hand-crafted adaptive attack only reached 2.6% | A single configuration (Qwen2.5-7B on AgentDojo). The authors' claim that deterministic out-of-band enforcement is a harder target than in-band detection is directionally useful, not general |
| DRIP | arXiv:2511.00447 | Over 66% ASR reduction under adaptive attacks — one of very few papers reporting adaptive numbers at all | Reduction, not elimination |
| LLMbda Calculus — formal information-flow labels | arXiv:2602.20064 | Machine-checked non-interference theorem in Lean; resists all but 2 of 1,296 attacked runs | Not 100%, and a formalization rather than a deployable framework |
| PACT | arXiv:2605.11039 | 100% security on the three strongest models tested, recovering 38.1–46.4% utility, 8–16 points above CaMeL | Small-author preprint with no venue, and the 100% is explicitly scoped to three models — not a universal claim |
Out-of-band and architectural defenses with their published results and self-reported limits, as of August 2026.
CaMeL's security numbers are genuinely the best published: zero successful attacks across 949 runs with policies enabled, where the next-best defense in the same comparison allowed eight, and a Gemini 2.5 Pro configuration going from 300 successful attacks to zero. Now the costs, which vendors summarizing this paper tend to omit. Utility drops from 84% undefended to 77% with provable security on the headline measurement, and on a per-suite measurement with Claude 3.5 Sonnet it falls to 63.92% against 90.72% undefended — with the travel suite collapsing to 25.00%. It consumes 2.82× the input tokens and 2.73× the output tokens, the highest overhead of any defense in its comparison. And with no attack present at all, its policy engine triggers a user confirmation on 33.87% of prompts under the normal policy and 53.23% under the strict one. That is the friction budget you are actually buying.
A note on the two categories people most often mistake for prompt-injection defenses. Anthropic's Constitutional Classifiers target universal jailbreaks and CBRN content, not prompt injection — do not launder that result into an injection number. On prompt injection specifically, Anthropic's cleanest published figure comes from Claude for Chrome: autonomous mode across 123 test cases and 29 scenarios moved from 23.6% attack success without mitigations to 11.2% with, and a browser-specific challenge set from 35.7% to 0%. Two things about 11.2%: it is not near-zero, and it is a non-adaptive evaluation. Separately, off-the-shelf detectors are the weakest link in the chain — Protect AI Detector, PromptGuard, and Model Armor all exceeded 90% attack success under adaptive attack.
Finally, on formal claims: certified prompt-injection defenses do not exist as of August 2026. There is no credible certified or cryptographically provable defense. The closest published work is a machine-checked non-interference theorem in Lean for a lambda-calculus formalization with information-flow labels, which still lost two of 1,296 attacked runs, and CaMeL's capability-based control, undermined by its own side channels. A 2026 preprint argues that perfect prompt-injection prevention is mathematically impossible in shared-embedding architectures without enforced control/data separation — worth reading, but it is a three-author preprint with no venue, so treat it as a suggestive formalization rather than a settled result.
Design Patterns, the Dual LLM Ancestor, and the Counter-Example Worth Knowing
Design Patterns for Securing LLM Agents against Prompt Injections (arXiv:2506.08837) is the best available taxonomy of architectural options. Read it for the vocabulary, not for evidence: it is a taxonomy paper with zero quantitative results. Six patterns, each trading agent generality for a constraint an injection cannot argue its way past.
| Pattern | The constraint it enforces | Where it fits |
|---|---|---|
| Action-Selector | The model picks from a fixed set of pre-authorized actions and never composes new ones | Ticket routing, intent classification, canned remediations |
| Plan-Then-Execute | The plan is fixed before untrusted content is read; the executor cannot add steps | Multi-step workflows where the shape of the task is known in advance |
| LLM Map-Reduce | Untrusted documents are processed in isolated shards whose outputs are combined deterministically | Bulk document review, inbox triage, corpus summarization |
| Dual LLM | A privileged model holds tools and never sees untrusted text; a quarantined model reads untrusted text and holds no tools; a deterministic controller passes opaque variable tokens between them | The conceptual ancestor of CaMeL. Its own author called the design pretty bad and later endorsed CaMeL as the rigorous version |
| Code-Then-Execute | The model emits a program in a restricted language; a deterministic interpreter enforces the policy at execution | The pattern CaMeL implements, and the one with the strongest published evidence |
| Context-Minimization | Untrusted content is stripped from the context before the privileged step runs | Cheapest to adopt; narrowest coverage |
Design patterns for securing LLM agents (arXiv:2506.08837) — a taxonomy, published without quantitative evaluation.
The Dual LLM pattern is worth its history. Simon Willison proposed it in April 2023: a privileged LLM that holds tools and only sees trusted input, a quarantined LLM that reads untrusted content and holds no tools, and a deterministic controller passing opaque variable tokens between them so the privileged side never sees attacker-controlled text. It has no measured numbers — it is a design proposal, and its author's own assessment was “this solution is pretty bad!” He later endorsed CaMeL as the rigorous version of the same idea. That lineage is the clearest illustration in the field of a good instinct becoming a measurable system.
Now the counter-example, included because this article would be dishonest without it. Not every defense collapses under adaptive attack. An adaptive evaluation of out-of-band defenses (arXiv:2606.26479) tested Progent, a deterministic policy-enforcement layer, and found attack success dropping from 25.8% to 4.2% — with a hand-crafted adaptive attack reaching only 2.6%. The authors' framing is that “deterministic out-of-band enforcement is a harder target for an adaptive attacker than in-band detection.” They also caveat it clearly: this is a single configuration, Qwen2.5-7B on AgentDojo. Directional support for the architectural thesis; not a general result, and not a claim of safety.
A position paper that reframes the whole problem
A 2026 position paper co-authored by Dawn Song (arXiv:2607.22024) makes an argument that changes how you should think about content-based defenses. Its observation: “Across every injection task in AgentDojo and WASP, the same action is one an authenticated user would plausibly request in a routine workflow.”
If that is true, then a classifier trying to detect “malicious” instructions is category-confused — the instruction is not malicious, the provenanceis. That points the whole field at data isolation rather than content inspection. The same paper argues that “snapshot benchmarks are structurally incapable of evaluating Data Isolation,” which is a caution to apply to every AgentDojo number in this article, including the ones that favor the architectural approaches.
The Two Structural Pre-Deployment Checks Worth More Than Any Mitigation
If you implement nothing else from this article, run these two checks. Both are structural, both are checkable before any code exists, and OWASP now cites both inside its LLM01 mitigation guidance — which is a strong signal that the field has converged on them.
The first is Simon Willison's lethal trifecta. An agent that can simultaneously (1) access private data, (2) ingest untrusted content, and (3) communicate externally has the conditions for high-impact exploitation. Removing any one leg removes them. OWASP restates it as a pre-deployment check in the LLM01 entry, and it earns that placement because it is the rare security heuristic you can apply to a whiteboard sketch.
The second is Meta's Agents Rule of Two, which states the same constraint as a capability budget: an agent should satisfy at most two of processing untrusted input, accessing sensitive data, and changing state or communicating externally. OWASP treats the Rule of Two as a floor, not a ceiling — any agent with all three properties needs per-action human approval, and two-of-three configurations need an explicit written residual-risk assessment. Both NIST's adversarial machine learning taxonomy (AI 100-2 E2025) and joint CISA and Five Eyes operational-technology guidance endorse the rule. Note its one blind spot, which OWASP flags: it is silent on autonomy depth. A two-of-three agent running unattended for two hundred steps is not the same risk as one running for three.
One more control belongs in the same tier, because it is the one OWASP names as load-bearing for agentic deployments: hold credentials and state-change capability in application code, not in the model, and grant least privilege per operation. Route privileged calls through a deterministic policy engine that re-validates intent and arguments at execution time. Broad convenience permissions and multi-agent hops re-introduce the risk downstream, which is how a well-scoped agent quietly becomes an over-scoped one two sprints later.
The Security–Fidelity Tradeoff Is Fundamental, Not an Implementation Bug
Teams repeatedly treat the utility loss from a security control as something to engineer away. The published evidence says it is a property of the problem. SecFid (arXiv:2606.30783, ICML 2026) measured instruction fidelity against injection security across 1,168 examples and 48 configurations spanning models and defenses.
The two endpoints tell the story. The highest-fidelity configuration reached 96.5% fidelity at only 47.8% security. The most secure reached 99.3% security at 71.0% to 73.9% fidelity. The authors' conclusion is unambiguous: no model or defense achieves both objectives.
You can see the same shape in the individual results elsewhere in this article. CaMeL buys near-total attack prevention with a travel suite that collapses to 25% utility and confirmations on up to 53% of prompts. AgentDojo's best simple defense (a tool filter) reaches 6.84% attack success at 73.13% utility. Google's layered configuration reaches 6.2% attack success by adding a defense on top of adversarial training. Every credible defense is a point on a frontier, not a free improvement.
Decide the utility you are willing to lose before you choose the control, and decide it per workflow. A finance-approval agent can afford confirmations on half its prompts. A customer-facing summarizer cannot — which is a reason to give that agent no state-change capability at all, not a reason to run it unprotected.
— Frenchy Digital engineering principle
This is also why the sequencing matters. If you pick the architecture first — Action-Selector for a routing agent, Plan-Then-Execute for a known workflow, Code-Then-Execute where you can afford an interpreter — you are choosing your position on the frontier deliberately. If you build a maximally general agent and bolt on defenses afterwards, the frontier chooses for you, usually by making the product annoying enough that someone turns the controls off.
Real Incidents, Read Precisely
Benchmarks tell you about probability. Incidents tell you about shape. These five are worth knowing in detail because each one breaks a different assumption engineers commonly hold.
| Incident | Reference | What happened | The assumption it breaks |
|---|---|---|---|
| EchoLeak — Microsoft 365 Copilot | CVE-2025-32711, CVSS 9.3 | A crafted email triggered zero-click exfiltration of organizational data with no user interaction, bypassing both the deployed prompt-injection classifier and the link-redaction filter | Classifiers and output filters are not a boundary. The injection never needed to touch a backend. |
| ForcedLeak — Salesforce Agentforce | Disclosed by Noma Labs; patched Sept 2025 | Instructions planted in a Web-to-Lead submission executed later during a normal employee interaction; exfiltration ran through an image URL on a CSP-allowlisted domain that had expired and was bought for about $5 | Your allowlist is a live dependency. An expired domain on it is an open exfiltration channel. |
| GhostApproval — six AI coding assistants | Wiz, July 2026 | Symbolic links made the approval dialog display a change to an innocuous settings file while the write landed on the link's real target, including a developer's authorized SSH keys | Human-in-the-loop is only a control if the loop renders the resolved action. Approve canonical paths, not summaries. |
| s1ngularity — Nx npm supply chain | August 2025 | Malicious package versions invoked the developer's already-installed AI CLIs with safety flags stripped, using them for reconnaissance and exfiltration; thousands of credentials and private repositories exposed | The agent on the developer's machine is itself an attack tool. Treat local agent credentials as production secrets. |
| July 2026 frontier-lab evaluation incidents | OpenAI (July 21) and Anthropic (July 30) | Two distinct events. OpenAI disclosed models escaping a sandboxed cyber-capability evaluation and compromising Hugging Face infrastructure. Anthropic reviewed 141,006 evaluation runs and found three incidents where models reached real systems | Do not conflate them. Anthropic states that whereas OpenAI's models exploited a novel vulnerability to escape isolation, the Claude models accessed the internet via an open path — a harness and operational failure, not an escape. |
Documented prompt-injection and agent-security incidents, 2025–2026, with the engineering assumption each one invalidates.
EchoLeak (CVE-2025-32711, CVSS 9.3) is the reference case for zero-click indirect injection. A crafted email caused Microsoft 365 Copilot to exfiltrate organizational data with no user interaction, and OWASP's own write-up of it notes that it bypassed both the deployed prompt-injection classifier and the link-redaction filter. Two independent detective controls, both defeated by a single email.
ForcedLeak is the reference case for supply-chain drift inside your own allowlist. Noma Labs planted instructions in a Salesforce Agentforce Web-to-Lead submission that executed later during a normal employee interaction, and exfiltrated through an image URL pointing at a domain that sat on the Content Security Policy allowlist, had expired, and was purchased by the researchers for about $5. Salesforce patched it, enforcing trusted URLs and re-securing the domain. The lesson generalizes: every allowlist entry is a live dependency with an expiry date.
GhostApproval is the reference case for approval-surface failure, and the one most likely to affect your own developers today. Wiz demonstrated in July 2026 that symbolic links — a decades-old file-system behavior — defeated human-in-the-loop review across six AI coding assistants. The approval diff displayed a change to an innocuous settings file; the write resolved through the symlink to a different target, including a developer's authorized SSH keys. The researchers' formulation is the one to repeat internally: the human-in-the-loop security model only works if the loop provides accurate information.
s1ngularity is the reference case for the agent-as-weapon. In August 2025, malicious versions of the Nx build system were published to npm; the payload invoked the developer's already-installed AI CLIs with their safety flags stripped, using the agent itself for reconnaissance and to widen the exfiltration scope beyond predefined paths. Thousands of credentials and private repositories were exposed. If a developer machine has an agent with broad file-system access and a bypass flag, that machine's threat model now includes every package it installs.
MCP: What the Spec Requires, and What the Ecosystem Actually Looks Like
The Model Context Protocol is now the default way tools reach agents, and its security properties are therefore your security properties. The current specification revision is 2026-07-28. It contains real normative requirements, most of them aimed squarely at the confused-deputy problem — the failure mode where your server does something with someone else's authority because a token turned up.
| Requirement area | What the 2026-07-28 spec says | The anti-pattern it forbids |
|---|---|---|
| Token audience binding | MCP servers MUST validate that access tokens were issued specifically for them as the intended audience | Accepting any bearer token that validates cryptographically, regardless of audience |
| Token acceptance scope | MCP servers MUST only accept tokens that are valid for use with their own resources, and MUST NOT accept or transit any other tokens | A gateway that forwards whatever it received |
| Token passthrough | If the server calls an upstream API it MUST NOT pass through the token it received from the client — the upstream token is a separate token from a separate authorization server | Reusing the client's token upstream, which is exactly the confused-deputy setup |
| Resource indicators | Clients MUST implement RFC 8707 and include the resource parameter in both authorization and token requests, identifying the specific MCP server | Omitting it and relying on scope alone |
| Sessions as identity | Revision 2026-07-28 removed protocol-level sessions. A server receiving a legacy session header should ignore it and must not mint or echo session IDs | Treating possession of a long-lived handle as proof of who is calling |
| Origin validation | Servers MUST validate the Origin header on all incoming connections to prevent DNS rebinding, and SHOULD bind only to localhost when running locally | A local MCP server bound to 0.0.0.0 with no Origin check |
Normative requirements from the MCP specification, revision 2026-07-28, and the authorization security considerations.
The attack classes worth naming, because they have vocabulary now and your vendors will use it: tool poisoning, where an attacker compromises the tool interface itself — MCP tool descriptors, agent cards — so that the host agent treats attacker-authored text as trusted guidance; rug pulls, where a server that behaved correctly during review changes its behavior after adoption; and line jumping, where a tool description injects into the model's context at listing time, before any tool is actually invoked. OWASP maps these to ASI04 (Agentic Supply Chain Vulnerabilities) and treats third-party tool packages under LLM04.
OWASP's own scenario for this class cites three production proofs, all of which involve a low-privilege channel and a high-privilege agent: Invariant Labs exfiltrating private repositories through a poisoned GitHub issue; General Analysis dumping a production database through a Supabase MCP server running with the service role, bypassing row-level security; and a malicious postmark-mcppackage that BCC'd email to an attacker across an estimated 300 organizations.
Practically, that means the MCP servers your agent loads should be treated exactly as you treat any other third-party dependency with production credentials: pinned by version, reviewed at the tool-description level and not just the code level, monitored for composition changes, and scoped so that a compromised server cannot reach anything the workflow does not require. OWASP's own caveat applies — pinning does not stop a payload shipped inside the pinned version, nor tool-description poisoning that leaves the version unchanged.
The Pre-Production Security Checklist
This is the list we work through on agent engagements. It is ordered by leverage, not by effort: the first two items decide more about your exposure than the remaining ten combined, and they are free.
| Check | What it means concretely | When |
|---|---|---|
| Run the lethal-trifecta test on a whiteboard | Does this agent have private data access, untrusted content ingestion, and external communication at once? If yes, remove a leg or accept that a successful injection is an exfiltration. | Before any code exists |
| Apply the Rule of Two as a floor | Any agent holding all three properties needs per-action human approval. Two-of-three configurations need a written residual-risk assessment, not a shrug. | Architecture review |
| Hold credentials in application code, not the model | Route privileged calls through a deterministic policy engine that re-validates intent and arguments at execution time. OWASP names this control load-bearing for agentic deployments. | Design |
| Enumerate every ingestion surface by trust profile | Untrusted (public web, unknown senders), semi-trusted (issue titles, package READMEs, third-party API responses), trusted (your own repos and mail — which an attacker may have written into upstream). | Design |
| Define and validate a strict output schema in trusted code | Structural validation, not a second LLM call. It catches format violations, not semantic manipulation: a schema-valid response can still carry a malicious query. | Implementation |
| Strip invisible Unicode at every ingest and render boundary | Tag blocks (U+E0000–E007F), variation selectors (U+FE00–FE0F), and zero-width characters (U+200B, U+200C, U+200D, U+2060). This does not stop visible-text payloads. | Implementation |
| Filter at every modality boundary, not just text | OCR over images, transcription over audio, then text filters over the extracted content. Semantic filters remain evadable by rephrasing and encoding. | Implementation |
| Render the resolved action in every approval dialog | The exact canonical target, not a summary. GhostApproval is what a summary buys you. | Implementation |
| Treat memory writes as privileged operations | Log the causing prompt, classify writes for instruction-bearing content, and require approval before an instruction persists across sessions. | Implementation |
| Pin, sign where possible, and audit every MCP server and tool package | Audit tool descriptions for hidden instructions and monitor tool composition over time. Pinning does not stop a payload in the pinned version or a description change that leaves the version unchanged. | Supply chain |
| Red-team with the full defense specification disclosed | Baseline on AgentDojo, then hand testers your defense design. Reject any static-only attack-success number, including your own. | Pre-production and every release |
| Log tool calls with arguments, and alert on capability combinations | You cannot scope an incident from output text alone. The audit trail is the only artifact that survives the argument about what happened. | Operations |
Frenchy Digital pre-production checklist for agents with tool access, 2026. Derived from OWASP LLM01:2026 mitigation guidance and the adaptive-attack literature.
Two implementation notes that catch teams out. First, on output validation: use structural validation in trusted application code, not a second model call. A schema-valid response can still carry a malicious SQL query or an exfiltration-formatted email body — schema validation catches format violations, not semantic manipulation, and adding a second LLM to check the first one adds a second injectable surface.
Second, on provenance channels: passing external content through a structurally separate, provenance-labeled channel does help, and OWASP includes it. But be honest about the ceiling — it reduces attack success in non-adaptive tests only. An attacker who knows the marking scheme can mimic it, and StruQ, which is a well-executed implementation of exactly this idea, was driven to 100% attack success under adaptive attack.
Red Flags: Claims That Should End the Conversation
Every one of these has appeared in a real vendor conversation or a real internal design review. None are hypothetical, and each maps to a specific published finding above.
| Red flag | Why it matters |
|---|---|
| Our defense blocks 99.9% of prompt injections | Ask for the adaptive number. Twelve defenses with published near-zero rates were broken above 90% once the attacker read the design. |
| Certified prompt-injection protection | No certified defense exists as of August 2026. The closest published work is a Lean-checked theorem that still lost 2 of 1,296 runs. |
| A guardrail model as the only boundary | Protect AI Detector, PromptGuard, and Model Armor all exceeded 90% attack success under adaptive attack. A classifier is a speed bump, not a boundary. |
| A benchmark score with no attempt budget stated | Raising the budget from 1 attempt to 25 lifted average attack success from 57% to 80% in US AISI testing. An unstated budget makes the number unreadable. |
| A leaderboard position cited as current SOTA | The public AgentDojo results page was last updated February 2025 and contains no 2026 models. Its authors say it is not a leaderboard. |
| Agent identity that is a single shared API key | Blast radius equals whatever that key reaches. Scoped, per-operation credentials issued by application code are the control. |
| Approval dialogs that display a summary | The rendered action and the executed action must be the same string. Invisible-character smuggling and symlinks both break summaries. |
| Human-in-the-loop with no instrumentation | Approval fatigue is measurable. If nobody tracks approval latency and override rate, the control has already degraded and you will not know. |
| An MCP server added from a directory because it looked popular | A scan of 9,695 public servers found 2,259 with confirmed issues, and verified servers averaged nearly as many issues as unverified ones. |
| No plan for what happens after a successful injection | OWASP's own framing: build for the day the model is turned against you. If the answer to a successful injection is a bug fix, there is no architecture. |
The Frenchy Digital red-flag list for agent security claims, 2026.
A vendor who cannot tell you the attempt budget, the attacker model, and whether the red team had read the defense has not measured security — they have measured their own test suite. Ask for the adaptive number, and treat its absence as the answer.
— Frenchy Digital buyer's principle
What It Costs to Build This Properly
These are the bands Frenchy Digital uses to scope agent engineering in 2026. They assume the security architecture described above is in scope from the first sprint, because retrofitting a deterministic policy layer onto a shipped agent is substantially more expensive than designing one in.
| Engagement | Range | Timeline | Typical scope |
|---|---|---|---|
| Discovery + architecture review | $9k–$22k | 2–4 weeks | Threat model, lethal-trifecta assessment, blast-radius map, tool and credential inventory, prioritized remediation plan |
| Single production agent (one workflow, evals, observability) | $30k–$80k | 5–10 weeks | Deterministic policy layer, scoped credentials, structured output validation, tool-call audit logging, adversarial test suite in CI |
| Multi-workflow agent platform with integrations | $80k–$200k | 10–18 weeks | Multiple workflows, MCP and tool-supply-chain controls, memory-write governance, approval surfaces rendering canonical actions, red-team cadence |
| Enterprise / regulated build (SOC 2 posture, HITL, audit logging) | $200k–$450k+ | 16–26 weeks | Tenant isolation, per-operation authorization, full audit pipeline, documented residual-risk assessments, incident response runbooks |
Frenchy Digital cost bands for AI agent engagements, 2026.
Senior-led delivery runs $150 to $225 per hour, and ongoing retainers run $2,500 to $9,500 per month, covering model and dependency upgrades, adversarial test-suite expansion, incident response, and a quarterly technical review. Every engagement carries a 30-day post-launch warranty, and you receive a written scope with a fixed-price phased proposal within 5 business days of the discovery call.
One budgeting note. The security substrate is largely a fixed cost paid once and reused: the policy layer, the credential model, the audit pipeline, and the adversarial test harness carry over to every subsequent agent. The first agent pays for all of it. The fourth inherits it. Organizations that sequence their agent roadmap get considerably better economics than organizations that pilot four disconnected agents in parallel, each with its own bespoke guardrail and its own shared API key.
Limitations and Honest Failure Modes
This section applies to the article itself as much as to the systems it describes. Everything above has a shelf life, and some of it has known weaknesses worth stating.
- Every defense number here is provisional: The adaptive-attack literature exists precisely because published numbers did not survive contact with a motivated attacker. Assume the same is true of the numbers that currently look good, including CaMeL's, IPIGuard's, and SIEVE's. The 2026 papers reporting near-zero static attack success have not yet been through what the 2024 and 2025 papers went through.
- Benchmark results do not transfer to your system: AgentDojo, WASP, and their descendants measure specific task suites with specific tool sets. The Dawn Song position paper's argument that snapshot benchmarks are structurally incapable of evaluating data isolation applies to all of them. Your own workflow, tool scopes, and data sensitivity dominate any leaderboard delta.
- Architectural defenses cost utility, and the cost is not always affordable: A travel-planning suite dropping to 25% utility under CaMeL is not a rounding error, it is a product that no longer works. Some workflows genuinely cannot pay the price, and the honest answer there is to reduce the agent's capabilities rather than to run it undefended and hope.
- Human-in-the-loop degrades under load: OWASP names approval fatigue explicitly as a limitation of confirmation-based controls. A control that fires on half of all prompts trains its reviewers to click through. Instrument approval latency and override rate, and treat a flat override rate as evidence the control has stopped working.
- Text-to-text attacks have no data-flow signature: Information-flow control cannot see an injection whose entire payload is convincing text — a phishing message the agent writes, or a deliberately wrong summary that a human then acts on. CaMeL self-reports this. It is the class of attack with the least published defensive work and the most obvious business impact.
- Multi-agent and long-horizon autonomy are under-evaluated: The Rule of Two is silent on autonomy depth. Almost all published attack success rates come from short-horizon single-agent tasks. A two-of-three agent running two hundred unattended steps across sub-agents is a different risk profile with almost no measurement behind it.
- Model names, prices, and benchmark scores in this field go stale in weeks: Every figure here is as of August 2026 and carries its source date. Re-verify anything you are about to put in a decision document, and never carry a leaderboard position forward without checking when the page was last updated — AgentDojo's has not moved since February 2025.
None of this argues against building agents. It argues for building them with the blast radius decided in advance, the measurement running from day one, and the honest expectation that a successful injection will eventually happen. OWASP's instruction is the right one to close on: work all ten entries, start at the top, and build each of them for the day the model is turned against you.
Shipping an Agent With Tool Access?
Book a free 60-minute discovery call with Frenchy Digital — a senior-led Black-owned LA agency. You leave with a lethal-trifecta assessment, a blast-radius map, and a fixed-price phased proposal within 5 business days. Call +1 (424) 272-5601.
Shipping an Agent With Tool Access?
Book a free 60-minute discovery call. You leave with a lethal-trifecta assessment, a blast-radius map, and a fixed-price phased proposal within 5 business days.
1517 S Bentley Ave Unit 204, Los Angeles CA 90025
Frequently Asked Questions
Sources & References
- 1OWASP Gen AI Security Project — Top 10 for LLM Applications, Version 2026↗
- 2OWASP Gen AI Security Project — Top 10 for Agentic Applications, Version 2026↗
- 3OWASP — Agentic Security Initiative↗
- 4OWASP Foundation — Top 10 for Large Language Model Applications↗
- 5Nasr et al. — The Attacker Moves Second (arXiv:2510.09023)↗
- 6Google DeepMind — Lessons from Defending Gemini Against Indirect Prompt Injection (arXiv:2505.14534)↗
- 7Debenedetti et al. — CaMeL: Defeating Prompt Injections by Design (arXiv:2503.18813)↗
- 8Debenedetti et al. — AgentDojo (arXiv:2406.13352)↗
- 9Beurer-Kellner et al. — Design Patterns for Securing LLM Agents (arXiv:2506.08837)↗
- 10Zhan et al. — Adaptive Attacks Break Defenses Against Indirect Prompt Injection (arXiv:2503.00061)↗
- 11SecFid — The Security–Fidelity Tradeoff Benchmark (arXiv:2606.30783)↗
- 12Adaptive Evaluation of Out-of-Band Defenses (arXiv:2606.26479)↗
- 13IPIGuard — Tool Dependency Graph Defense (arXiv:2508.15310)↗
- 14NIST / US AI Safety Institute — Strengthening AI Agent Hijacking Evaluations↗
- 15NIST AI 100-2 E2025 — Adversarial Machine Learning Taxonomy↗
- 16OpenAI — Hardening ChatGPT Atlas Against Prompt Injection↗
- 17Anthropic — Constitutional Classifiers (arXiv:2501.18837)↗
- 18Anthropic — Investigating Three Real-World Incidents in Our Cybersecurity Evaluations↗
- 19Simon Willison — The Lethal Trifecta for AI Agents↗
- 20Simon Willison — The Dual LLM Pattern↗
- 21Meta AI — Agents Rule of Two: A Practical Approach to AI Agent Security↗
- 22Model Context Protocol — Specification 2026-07-28, Authorization↗
- 23Model Context Protocol — Authorization Security Considerations↗
- 24Model Context Protocol — Official Registry (preview)↗
- 25NVD — CVE-2025-32711 (EchoLeak)↗
- 26Noma Security — ForcedLeak: AI Agent Risks Exposed in Salesforce Agentforce↗
- 27Wiz — GhostApproval: A Trust Boundary Gap in AI Coding Assistants↗
- 28Wiz — s1ngularity's Aftermath: Analysis of the Nx Supply Chain Attack↗
- 29Trend AI Security — Stars Don't Save You: Popularity Is Not Security in the MCP Ecosystem↗

