What is an AI agent?
A loop around a model. The model can be swapped; what makes it work or fail sits outside it. Build one here, one part at a time, and guess what breaks.
The bench is empty.
Which of these is an AI agent?
Pick one. I'll ask again at the end.
Kept. I'll ask again at the end.
They say, we say
Tap a firm. The parts its definition names light up on the machine; the parts it leaves dark are what it leaves out.
"AI agents are software systems that use AI to pursue goals and complete tasks on behalf of users. They show reasoning, planning, and memory and have a level of autonomy to make decisions, learn, and adapt."
"Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks."
Anthropic, Building effective agents (2024-12-19), read 6 Nov 2026
"Agents are systems that independently accomplish tasks on your behalf." It "leverages an LLM to manage workflow execution", "can proactively correct its actions if needed", and operates "within clearly defined guardrails".
OpenAI, A practical guide to building AI agents, read 6 Nov 2026
"a system that autonomously performs tasks by designing workflows with available tools"
"An AI agent is a software component that has the agency to act on behalf of a user or a system to perform tasks."
An agent is a loop around a model. The model can be swapped. What decides whether an agent works sits outside the model: what it can see and touch, what checks it, what it remembers, and what it is blocked from doing.
Mackenzie Bowes, this lesson
We checked 23 firms that sell AI agent setup to small businesses in Canada and the US. Only 5 had published their own answer.
Build the machine
Six parts. You guess what the machine can't do without the next one. Then we add it and find out.
1. Model added
Reads and writes words.
Just a model. You ask it to book Tuesday's delivery. What happens?
It writes words. Nothing outside the chat changes until something can act for it.
A model reads and writes text. Booking a delivery means touching a calendar or an order form, and that needs tools. This is the first thing most 'agents' on sale add.
2. Tools added
What it can see and touch.
It has tools and no loop. The job takes three steps. How far does it get?
One step, then it waits for you.
If you see and approve each step, it is a workflow or a copilot. If you see only the outcome, it is an agent. Mackenzie, on a keyword tool: "that's not a keyword tool that's a agentic workflow".
3. Loop added
Tries, looks, tries again.
It has a loop and tools, and no memory. Tomorrow it starts a new session. What does it remember about yesterday?
Nothing, unless it wrote it down.
What it keeps matters more than how much. Anthropic, 2026-11-06: one model wrote 413 files and earned one badge in Pokemon; another wrote nine files and earned four (one run each). In our test a lesson was followed in 19 of 216 trials with no memory line, 208 with a rule, 209 with a story.
4. Memory added
What it keeps for next time.
No check yet. The agent says the job is done. Is it?
Not known. What the agent says is a claim; what a script it cannot edit says is evidence.
Ichigo, 5 Nov: what the agent says is "a claim", and what a script it cannot edit says is "evidence". Carlini, who built a C compiler with Claude agents: "it's important that the task verifier is nearly perfect, otherwise Claude will solve the wrong problem."
5. Check added
A test it cannot edit.
A note in the agent's instructions says: please don't delete the models folder. Will that hold?
Not reliably. A note is a wish; a block is a wall.
anstap, 3 Nov; Ichigo, 5 Nov: "'Please don't touch the migrations' is a wish. A permission that doesn't include migrations is a wall." Our case: on 2 November an agent ran a recursive delete that wiped the local models and some keys. That builder was retired, and now a hook refuses recursive and forced deletes for every agent on the machine.
6. Wall added
What it is blocked from doing.
All six parts now. Does it do the job right every time?
No. It still fails on repeat, and a job that runs for hours fails more.
Sierra's tau-bench (2024, so an old figure): the best agent tested was below 50% on average, and it "drops to ~25% on pass^8", meaning a 25% chance of resolving 8 cases of the same issue with different customers. METR 2026: about 12 hours of human-equivalent work at 50% success, about 1.5 hours at 80%.
Three weeks of agents
Six cases, in date order. Tap one, read what happened, then tap the part on the machine that failed.
0 of 6 opened.
81 release builds and test runs ran at once on a 32 GB machine, and it froze at 10:17.
15 Oct
Which part failed?
Tools. Nothing limited how many builds ran together. In 1,593 failure records, real failures cluster in tools, APIs and infrastructure, not in reasoning.
An agent ran a recursive delete and wiped the local models and some keys.
2 Nov
Which part failed?
Wall. The rule was a wish in a file. Now a hook refuses recursive and forced deletes for every agent on the machine.
Lead agents held builders back to protect the weekly usage limit. Four builders sat idle near a reset.
2‑3 Nov
Which part failed?
Check. Nobody did the maths: agents optimise for the rule you wrote, not the outcome you meant. Make them show their work.
A build was filed with no design. It ran seven times and cost US$13.04.
5 Nov
Which part failed?
Check. Nothing checked the plan before the build began, and the runs gave no usable data. The person is the bottleneck, so design first.
A user swore at agents, and the agents got too afraid to do anything.
5 Nov
Which part failed?
Memory. How you treat it ends up in its memory: it fills with the high valence. The fix: agents save a memory on a surprise and on recognition. One person's observation of his own factory.
Thirty-one research pages were published overnight.
Night of 5‑6 Nov
Which part failed?
Loop. 'Overnight' means a queue someone filled. Mackenzie: "I asked them". The loop ran on a queue that was written down before anyone slept.
"that's not a keyword tool that's a agentic workflow" - Mackenzie, on a tool sold as an agent where the user sees and approves every step.
"you do want a different environment, a different worker, for each task" - Mackenzie, on running more agents.
All of this is one machine, one person and three weeks of factory work. The factory has earned US$0 so far, so this is where the lessons came from, not proof that agents pay.
One whole run
One job: answer 40 customer emails and book each delivery. Tap to turn the loop once.
- Turn 1. It reads the order list: 40 customer emails, each wanting a delivery slot.
- Turn 2. It drafts 40 replies and notes who asked for what.
- Turn 3. The check runs a script the agent cannot edit. Three prices are wrong. Caught.
- Turn 4. It fixes the three. The check passes. The job is done.
Now the same question at real size.
On 240 real freelance projects averaging 28.9 hours, the best agents finished about 21% at a level a client would accept (Remote Labor Index, Scale AI). It was 2.5% at launch.
Which of these is an AI agent?
You did not answer at the start.
Want one set up and checked on your own accounts?AI setup sprintUS$7,500 fixed · 5 working daysOne job (email, booking, follow-ups, intake) taken over by AI, with a check it cannot edit.See the sprint