Do AI agents actually work? What four benchmarks and one analyst forecast measured between 2024 and 2026
Benchmarks from 2024 to 2026 show agents finishing some tasks and few of them every time. Gartner expects over 40 percent of agentic AI projects to be canceled by the end of 2027. Read on 2026-11-06.
Contents
On the tests read for this page, AI agents do complete a share of real-looking work tasks, and almost none complete the same task reliably across repeated tries. The best scores are a third to a half of tasks on measures that differ from test to test. Separately, the analyst firm Gartner forecasts that over 40 percent of agentic AI projects will be canceled by the end of 2027. Each source below is listed with who built it, what it measured and what it cannot tell you.
What the tests measured
- τ-bench (Yao, Shinn, Razavi and Narasimhan, submitted 17 June 2024). Agents hold simulated customer conversations with tools and company rules, and the final database state is compared with the goal state.[1] The authors report that "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)."[1] They introduced "a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials", so pass^8 asks whether an agent gets the same task right on eight tries.[1] The model tested is a 2024 one.
- TheAgentCompany (Xu and colleagues at Carnegie Mellon University, submitted 18 December 2024, revised 10 September 2025). Agents work inside a simulated small software company with internal web sites, browsing, coding and messaging coworkers.[2] The authors "find that the most competitive agent can complete 30% of tasks autonomously", and conclude that "a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems."[2]
- AccountingBench (Penrose, 2025). Agents close a real software business's books month after month, against a CPA's baseline.[3] "Grok 4 and Claude 4 tend to perform well initially (within 1% of CPA baselines), but accumulate material errors over time", and three other models "were unable to close 1 month of books, giving up partway through."[3] Penrose built the test.
- APEX-Accounting (Mercor with Ramp, July 2026). 160 tasks written and solved by accounting experts, graded on expert rubrics.[4] Across nine frontier models the leader scores 56.4 percent on the mean-criteria measure, "No model scores more than 2.6% Pass^8", and the highest Pass@8 is 21.5 percent.[4]
Read together, these are four separate measures: share of tasks finished (τ-bench, TheAgentCompany), months of work before errors build up (AccountingBench), and share of expert criteria met and repeat reliability (APEX-Accounting). The scores cannot be added or averaged, and the models differ by test and year.
What the analyst forecast says
Gartner's press release of 25 June 2025 predicts: "Over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls."[5] It says many vendors take part in "agent washing", rebranding "existing products, such as AI assistants, robotic process automation (RPA) and chatbots", and that "only about 130 of the thousands of agentic AI vendors are real."[5] It adds: "Many use cases positioned as agentic today don't require agentic implementations."[5] Where IBM, Anthropic and Gartner each draw the line between an agent and a chatbot is on AI agent vs chatbot: where the line is drawn. The release cites a January 2025 poll of 3,412 webinar attendees: 19 percent said their organization had made significant investments in agentic AI and 42 percent conservative ones.[5] A forecast is an estimate, and the release gives no method for it. The Gartner forecast set beside McKinsey's survey and a vendor-run ROI survey is on Agentic AI: ROI figures and what they count.
What a buyer can take from this
Synthesis, this page's own reading and not any source's claim: ask a vendor for results over repeated runs of the same task, not a single demo, because τ-bench and APEX-Accounting both show the gap between one success and eight. Ask what share of tasks finishes without a person stepping in, and who checks the rest. Ask whether a plain rule-based tool would do the job, since Gartner says many use cases do not need an agent.
Not yet proven
- That benchmark scores predict results in a small business. All four tests use simulated or sampled work, and none measures agents deployed in a small firm.
- That these scores hold for current models. The tested models range from 2024 to 2026, and the best scores come from different tests.
- That the cancellation forecast will come true. It is a prediction, and the release states no method.
- How often agents finish a task every time in practice. Only τ-bench and APEX-Accounting report repeat-run reliability, in two domains.
If you are running an agent in a business and have counted how often it finishes a task on the first try, I would like to compare notes: mackenzie@mackenziebowes.com.
Sources
- Yao, Shinn, Razavi and Narasimhan, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv 2406.12045, submitted 2024-06-17 (abstract page). arxiv.
org/ abs/ 2406. 12045 - Xu et al. (Carnegie Mellon University), TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks, arXiv 2412.14161, submitted 2024-12-18, revised 2025-09-10 (abstract page). arxiv.
org/ abs/ 2412. 14161 - Penrose, AccountingBench: Evaluating LLMs on Real Long-Horizon Business Tasks (2025; the page carries no date). accounting.
penrose. com/ - Benchek et al. (Mercor), APEX-Accounting, arXiv 2607.27189, submitted 2026-07-29, revised 2026-07-30 (abstract page). arxiv.
org/ abs/ 2607. 27189 - Gartner, Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027, press release, 2025-06-25. www.
gartner. com/ en/ newsroom/ press- releases/ 2025- 06- 25- gartner- predicts- over- 40- percent- of- agentic- ai- projects- will- be- canceled- by- end- of- 2027
Terms on this page
- Agentic AI: AI that works toward a goal by planning steps, making decisions and acting with more or less independence.
- AI agent: A system where a language model decides for itself how to do a task and which tools to use.
- Tool call: A request from an AI model to run one action, such as a search or a file write, which the surrounding program carries out.