Do AI agents actually work? What four benchmarks and one analyst forecast measured between 2024 and 2026

Benchmarks from 2024 to 2026 show agents finishing some tasks and few of them every time. Gartner expects over 40 percent of agentic AI projects to be canceled by the end of 2027. Read on 2026-11-06.

By Mackenzie Bowes · mackenzie@mackenziebowes.com · @_mackenziebowes on X

Contents

On the tests read for this page, AI agents do complete a share of real-looking work tasks, and almost none complete the same task reliably across repeated tries. The best scores are a third to a half of tasks on measures that differ from test to test. Separately, the analyst firm Gartner forecasts that over 40 percent of agentic AI projects will be canceled by the end of 2027. Each source below is listed with who built it, what it measured and what it cannot tell you.

What the tests measured

Read together, these are four separate measures: share of tasks finished (τ-bench, TheAgentCompany), months of work before errors build up (AccountingBench), and share of expert criteria met and repeat reliability (APEX-Accounting). The scores cannot be added or averaged, and the models differ by test and year.

What the analyst forecast says

Gartner's press release of 25 June 2025 predicts: "Over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls."[5] It says many vendors take part in "agent washing", rebranding "existing products, such as AI assistants, robotic process automation (RPA) and chatbots", and that "only about 130 of the thousands of agentic AI vendors are real."[5] It adds: "Many use cases positioned as agentic today don't require agentic implementations."[5] Where IBM, Anthropic and Gartner each draw the line between an agent and a chatbot is on AI agent vs chatbot: where the line is drawn. The release cites a January 2025 poll of 3,412 webinar attendees: 19 percent said their organization had made significant investments in agentic AI and 42 percent conservative ones.[5] A forecast is an estimate, and the release gives no method for it. The Gartner forecast set beside McKinsey's survey and a vendor-run ROI survey is on Agentic AI: ROI figures and what they count.

What a buyer can take from this

Synthesis, this page's own reading and not any source's claim: ask a vendor for results over repeated runs of the same task, not a single demo, because τ-bench and APEX-Accounting both show the gap between one success and eight. Ask what share of tasks finishes without a person stepping in, and who checks the rest. Ask whether a plain rule-based tool would do the job, since Gartner says many use cases do not need an agent.

Not yet proven

If you are running an agent in a business and have counted how often it finishes a task on the first try, I would like to compare notes: mackenzie@mackenziebowes.com.

Sources

  1. Yao, Shinn, Razavi and Narasimhan, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv 2406.12045, submitted 2024-06-17 (abstract page). arxiv.org/abs/2406.12045
  2. Xu et al. (Carnegie Mellon University), TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks, arXiv 2412.14161, submitted 2024-12-18, revised 2025-09-10 (abstract page). arxiv.org/abs/2412.14161
  3. Penrose, AccountingBench: Evaluating LLMs on Real Long-Horizon Business Tasks (2025; the page carries no date). accounting.penrose.com/
  4. Benchek et al. (Mercor), APEX-Accounting, arXiv 2607.27189, submitted 2026-07-29, revised 2026-07-30 (abstract page). arxiv.org/abs/2607.27189
  5. Gartner, Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027, press release, 2025-06-25. www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027

Terms on this page

  • Agentic AI: AI that works toward a goal by planning steps, making decisions and acting with more or less independence.
  • AI agent: A system where a language model decides for itself how to do a task and which tools to use.
  • Tool call: A request from an AI model to run one action, such as a search or a file write, which the surrounding program carries out.