Can you stop an AI agent being tricked?
OpenAI says prompt injection is 'unlikely to ever be fully solved'. In one study, adaptive attacks got past 12 published defences, most at over 90 percent. OWASP and Meta advise limiting what the agent can do; no source measures it. Read 2026-11-06.
Contents
Short answer, from the sources below: nobody who has published on this claims a filter that reliably stops an AI agent from being tricked by text it reads, and two papers report filters being got round at high rates. The advice that OWASP, OpenAI and Meta give in common is about what the agent is allowed to do: fewer tools, narrower permissions, a person approving the risky step. No source here tests these controls on a small business's own agent.
What the risk is
An agent reads text (an email, a web page, a document) and acts on it. If that text contains instructions, the model may follow them as if the owner had written them. OWASP, the security-practice nonprofit, calls this prompt injection and says it can "enable unauthorized access" or "influence critical decisions". On whether it can be prevented, its page says: "it is unclear if there are fool-proof methods of prevention for prompt injection."[1]
OWASP separates a second problem, which is what makes an injection costly: "Excessive Agency", where the agent has more power than the job needs. Its page names three root causes: "excessive functionality", "excessive permissions" and "excessive autonomy".[2] Its example of excessive permissions is an extension meant to read data that "connects to a database server using an identity that not only has SELECT permissions, but also UPDATE, INSERT and DELETE permissions."[2]
EchoLeak (CVE-2025-32711) was, in its authors' words, "a zero-click prompt injection vulnerability in Microsoft 365 Copilot that enabled remote, unauthenticated data exfiltration via a single crafted email."[5] The US National Vulnerability Database record describes it as "Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network", and lists a score of 9.3 CRITICAL from Microsoft and 7.5 HIGH from NIST.[6] The paper reports that the attack went through by "evading Microsofts XPIA (Cross Prompt Injection Attempt) classifier", which is a guardrail of the kind described below.[5]
What a guardrail is and what the products offer
A guardrail is a check that runs on what goes into the agent or comes out of it. OpenAI's Agents SDK documents three placements: input guardrails, output guardrails and tool guardrails. Its page says "Input guardrails run only for the first agent in the chain" and "Output guardrails run only for the agent that produces the final output", so a chain of several agents is only partly covered unless tool guardrails are added.[9] The default mode runs the check alongside the agent to save time, and the page warns that "if the guardrail's tripwire is triggered, the agent may have already consumed tokens and executed tools before being cancelled."[9]
OpenAI's guardrails library page lists the checks on offer, with an estimated delay for each. Its jailbreak check "Detects attempts to jailbreak LLM calls via role-playing, system prompt overrides and injections", at an estimated 1,000 ms. A check for injection in tool calls, labelled Experimental, "Detects when function calls or function call outputs are not aligned with the user prompt's intent using LLM-based analysis."[10] Most of its checks are run by a language model, which can be tricked as the agent can. The page gives no detection rates.
Four other guardrail tools and what each says it does
- NeMo Guardrails (NVIDIA, Apache-2.0) is "an open-source toolkit for easily adding programmable guardrails to LLM-based conversational applications". Its readme says it protects against "jailbreaks and prompt injections", lists input rails that "can reject the input" or alter it, and includes jailbreak and injection detection. It also says: "Evaluating the safety of a LLM-based conversational application is a complex task and still an open research question."[12]
- Guardrails AI (Apache-2.0) is a Python framework whose Input/Output Guards "detect, quantify and mitigate the presence of specific types of risks", built from validators on its Guardrails Hub. Its readme names no prompt-injection check; it points to the Hub for the list. A notice dated 6 July 2026 says "Guardrails is discontinuing its hosted remote inferencing", with a cutoff set for 25 August 2026 (now past).[13]
- Llama Guard 4 (Meta, read on NVIDIA's hosted model card) is "a 12-billion parameter, dense, multimodal safety classifier". It labels a prompt or a response safe or unsafe against 14 hazard categories, from S1 Violent Crimes to S14 Code Interpreter Abuse. None of the categories is named prompt injection. The card says Llama Guard 4 "may be susceptible to adversarial attacks or prompt injection attacks" and points to Llama Prompt Guard 2 for detecting them; Hackett's abstract names "Meta's Prompt Guard" among the detectors it evaded, without a version number.[14]
- Anthropic's safeguards for its Claude browser extension are training against injections, classifiers that scan untrusted content, and human red teaming. Anthropic reports a 1 percent attack success rate for Claude Opus 4.5 against its own adaptive attacker, "given 100 attempts per environment", and writes: "No browser agent is immune to prompt injection". That figure is the vendor's own measurement against its own attacker, so it is not comparable with the 90 percent figures above.[15]
How well do guardrails hold up?
Two studies attacked them directly.
- Evasion of commercial detectors. Hackett and colleagues tested "six prominent protection systems, including Microsoft's Azure Prompt Shield and Meta's Prompt Guard", and report "achieving in some instances up to 100% evasion success" using character tricks and machine-learning methods.[4] This is a 2025 preprint to be published at a workshop, and it tests detectors on their own, not whole agents.
- Adaptive attackers. A team from Google DeepMind, ETH Zurich and others argues that defences were judged against weak, fixed attacks. They tuned general attack methods against each defence and report: "we bypass 12 recent defenses", with "attack success rate above 90% for most". They add that "the majority of defenses originally reported near-zero attack success rates."[3] The gap between a defence's own report and an attacker who adapts to it is the finding.
A January 2026 survey of 78 studies on coding assistants reports that attack success rates against state-of-the-art defences "exceed 85% when adaptive attack strategies are employed."[11] That is a summary of other people's results, so it adds no new measurement.
The vendors say similar things in their own words. OpenAI's post on its Atlas browser agent says prompt injection "is unlikely to ever be fully" solved, comparing it to scams aimed at people, and describes an attack its own automated attacker found: a malicious email led the agent, which was asked to draft an out-of-office reply, to send a resignation letter, because it "treats the injected prompt as authoritative, and follows it."[7] Meta's page calls prompt injection a "fundamental, unsolved weakness in all LLMs".[8]
What the sources suggest instead
Meta's "Agents Rule of Two" is a design rule, not a filter. It says that "until robustness research allows us to reliably detect and refuse prompt injection, agents must satisfy no more than two of the following three properties within a session": [A] the agent can process untrustworthy inputs, [B] it can have access to sensitive systems or private data, [C] it can change state or communicate externally.[8] If a job needs all three, "the agent should not be permitted to operate autonomously" and needs supervision, such as human approval.[8] A mailbox assistant that reads incoming email (A), sees the whole inbox (B) and can send mail (C) hits all three.
OWASP's lists point the same way. Against excessive agency it advises minimising the extensions an agent can call, avoiding open-ended ones, minimising their permissions, running actions in the user's own context, and requiring approval. It adds that downstream systems should "implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed."[2] Against injection it lists constraining the model's behaviour, least-privilege access, and "Implement human-in-the-loop controls for privileged operations to prevent unauthorized actions."[1]
These are recommendations from standards bodies and vendors. None of the sources measures how many injection attacks they stop, or what they cost in staff time when a person must approve each step.
You can't narrow what you haven't written down. The AI setup sprint sets up one job on your own accounts and hands you a sheet of every piece of client data it touches, who can see it and which AI model reads it.
Not yet proven
- How often these attacks work against a small business's own agent. The attack rates above come from research settings and from vendor products. No published measurement covers a custom agent built in a no-code tool or a framework.
- Whether the least-privilege advice reduces harm in practice. OWASP, OpenAI and Meta recommend it. No study in these sources compares agents with and without it.
- Detection rates of current guardrail products. OpenAI's page and NeMo's readme link to evaluations of their own checks, and this page quotes neither. No independent test of the 2026 versions of these tools is among the sources. Hackett's study tests 2025 detectors.
- Whether NeMo, Guardrails AI and Llama Guard stop prompt injection. NeMo's readme claims jailbreak and injection detection. Guardrails AI's readme and Llama Guard's category list name no such check. None of the three pages gives a detection rate.
- An outside check of Anthropic's 1 percent figure. It comes from Anthropic's own attacker, with the chart behind it not reproduced here.
- The cost of human approval. No source reports how often approval prompts are clicked through without being read, or how much time they take.
- Whether the figures are current. The two attack papers are from 2025, and defences have changed since. No source here reports a 2026 attack test of a named guardrail product.
If you run an agent that reads outside email or web pages and has measured what happens when someone tries to trick it, I would like to compare notes: mackenzie@mackenziebowes.com.
Sources
- OWASP Gen AI Security Project, LLM01:2025 Prompt Injection. Read in Rotunda 2026-11-06. genai.
owasp. org/ llmrisk/ llm01- prompt- injection/ - OWASP Gen AI Security Project, LLM06:2025 Excessive Agency. Read in Rotunda 2026-11-06. genai.
owasp. org/ llmrisk/ llm062025- excessive- agency/ - Nasr et al., The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections, arXiv 2510.09023, 10 October 2025 (abstract). Read in Rotunda 2026-11-06. arxiv.
org/ abs/ 2510. 09023 - Hackett et al., Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systems, arXiv 2504.11168 v3, 14 July 2025 (abstract). Read in Rotunda 2026-11-06. arxiv.
org/ abs/ 2504. 11168 - Reddy and Gujral, EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System, arXiv 2509.10540, 6 September 2025 (abstract). Read in Rotunda 2026-11-06. arxiv.
org/ abs/ 2509. 10540 - NIST National Vulnerability Database, CVE-2025-32711. Read in Rotunda 2026-11-06. nvd.
nist. gov/ vuln/ detail/ cve- 2025- 32711 - OpenAI, Continuously hardening ChatGPT Atlas against prompt injection attacks. Read in Rotunda 2026-11-06. openai.
com/ index/ hardening- atlas- against- prompt- injection/ - Meta, Agents Rule of Two: A Practical Approach to AI Agent Security, 31 October 2025. Read in Rotunda 2026-11-06. ai.
meta. com/ blog/ practical- ai- agent- security/ - OpenAI Agents SDK documentation, Guardrails (v0.24.0). Read in Rotunda 2026-11-06. openai.
github. io/ openai- agents- python/ guardrails/ - OpenAI Guardrails, list of input, output and agentic guardrails. Read in Rotunda 2026-11-06. guardrails.
openai. com/ - Maloyan and Namiot, Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems, arXiv 2601.17548, 24 January 2026, a survey of 78 studies. Read in Rotunda 2026-11-06. arxiv.
org/ html/ 2601. 17548v1 - NVIDIA NeMo Guardrails library, GitHub readme (release 0.24.1). Read in Rotunda 2026-11-06. github.
com/ NVIDIA- NeMo/ Guardrails - Guardrails AI, GitHub readme. Read in Rotunda 2026-11-06. github.
com/ guardrails- ai/ guardrails - NVIDIA Build, model card for Meta's Llama Guard 4 12B. Read in Rotunda 2026-11-06. build.
nvidia. com/ meta/ llama- guard- 4- 12b/ modelcard - Anthropic, Mitigating the risk of prompt injections in browser use. A vendor's own results. Read in Rotunda 2026-11-06. www.
anthropic. com/ research/ prompt- injection- defenses
Terms on this page
- AI agent: A system where a language model decides for itself how to do a task and which tools to use.
- Human in the loop: A person takes part in an automated system's work, supervision or decisions.
- Prompt injection: Text that makes an AI model do something its owner did not intend.
- Tool call: A request from an AI model to run one action, such as a search or a file write, which the surrounding program carries out.