How to test an AI agent and whether retrieval cuts hallucinations
Anthropic separates what an agent says from the state it leaves behind, and a task passed once from a task passed every time: 75 percent per trial is 42 percent over three. Retrieval cut hallucinations in legal AI tools but left 17 to 33 percent. Read 2026-11-06.
Contents
Testing an agent means more than checking one answer. This page sets out how one lab describes agent tests and what two measurements say about hallucination when an AI is given documents to work from. It reads Anthropic's engineering article on agent evals, a preregistered study of legal AI tools, Vectara's public leaderboard and two vendors' hallucination checks. It sits beside Do AI agents actually work? and How AI agents fail, which cover what agents do on benchmarks and how they break.
How to test an agent
Anthropic's article defines the parts. "A grader is logic that scores some aspect of the agent's performance", and a task can have several. A transcript is the full record of a run, while "The outcome is the final state in the environment at the end of the trial": an agent can say a flight is booked when the test that counts is "whether a reservation exists in the environment's SQL database."[1] "Agent evaluations typically combine three types of graders: code-based, model-based, and human", and model-based graders need calibration against human ones.[1]
Two kinds of test are kept apart. Capability evals ask what the agent can do; regression evals ask whether it still does what it did, and "should have a nearly 100% pass rate."[1] Reliability gets its own metric: "pass@k measures the likelihood that an agent gets at least one correct solution in k attempts", while "pass^k measures the probability that all k trials succeed." The article's example: "If your agent has a 75% per-trial success rate and you run 3 trials, the probability of passing all three is (0.75)³ ≈ 42%."[1] For an agent that talks to customers, it is the second number a business lives with.
Does retrieval reduce hallucinations?
Retrieval-augmented generation (RAG) gives the model documents to answer from. A preregistered study by Magesh and colleagues tested legal research tools that use it, whose makers had described retrieval as eliminating or avoiding hallucinations. It reports: "While hallucinations are reduced relative to general-purpose chatbots (GPT-4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time."[2] The authors conclude "the providers' claims are overstated."[2] The result is from 2024, for legal questions and three named products.
Vectara measures a different job from the legal study: summarizing a supplied document, with no research or citing involved, so its percentages and the legal study's are not one scale. Its public leaderboard, "last updated on October 22, 2026", "evaluates how often an LLM introduces hallucinations when summarizing a document", scored by Vectara's own detection model, HHEM.[3] Hallucination rates in the table run from 1.8 percent at the top to 24.2 percent at the bottom; OpenAI's gpt-6-sol is listed at 6.5 percent and Google's gemini-2.5-pro at 7.0 percent.[3] So even with the document in front of it, every model listed gets some summaries wrong.
What hallucination checks measure
The checks below test a reply against the documents the system was given, not against the world.
- Ragas faithfulness "measures how factually consistent a response is with the retrieved context", as the share of the response's claims the context supports; it can also be run with Vectara's HHEM model.[4]
- OpenAI's Guardrails library lists a hallucination detection check that "Validates claims against actual documents and flags potentially fabricated information", with an estimated delay of 8,000 ms.[5]
- Vectara's HHEM is the model behind the leaderboard above.[3]
By their definitions, a reply that agrees with a wrong source document is not flagged by any of the three.
Not yet proven
- Whether RAG helps outside law. The legal study covers legal research tools and one date.
- How well the detection checks work. None of the pages read reports how often a check misses a hallucination or flags a correct answer.
- Leaderboard independence. Vectara scores the table with its own model, so the ranking inherits that model's errors.
- A small business's agent. The evals article describes practice at Anthropic and its customers; it reports no results for a small business's own agent.
- Which metrics predict customer harm. No source links a pass^k figure to complaints or losses.
If you test an agent in your business and keep the failures, I would like to compare notes: mackenzie@mackenziebowes.com.
Sources
- Anthropic, Demystifying evals for AI agents. Read in Rotunda 2026-11-06. www.
anthropic. com/ research/ demystifying- evals- for- ai- agents - Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, arXiv 2405.20362, 30 May 2024 (abstract). Read in Rotunda 2026-11-06. arxiv.
org/ abs/ 2405. 20362 - Vectara, Hallucination Leaderboard (GitHub readme), last updated October 22, 2026. A vendor's own benchmark. Read in Rotunda 2026-11-06. github.
com/ vectara/ hallucination- leaderboard/ - Ragas documentation, Faithfulness. Read in Rotunda 2026-11-06. docs.
ragas. io/ en/ latest/ concepts/ metrics/ available_metrics/ faithfulness/ - OpenAI Guardrails, list of input, output and agentic guardrails. Read in Rotunda 2026-11-06. guardrails.
openai. com/
Terms on this page
- AI agent: A system where a language model decides for itself how to do a task and which tools to use.
- Retrieval-augmented generation: A language model that looks up outside text and uses it when it writes an answer.