The one file that makes your agent read what came after its training
11 to 2. That's the handwritten skill vs. the one Opus wrote.
A wall shelf is split by a post, and a dashed line runs from the post down to the floor: books the agent was trained on to the left, faint books published since to the right. The agent walks up to the line and stops, its lamp dark. A row of ten lamps records 1 run that went past: no skill, 1 run in 10. A page, the do-research skill, arcs into its hand, the lamp lights green, and the agent walks past the line, each book it passes turning green. A second row records 8: with do-research, 8 runs in 10.
Contents
Ask Claude Haiku to research something and count what happens. One run in ten, it goes to the web. The other nine, it answers from memory - and here's the unsettling part: it sounds exactly as confident either way.
Your agent's training stopped months ago. People kept publishing. The agent doesn't know what it hasn't read, so it doesn't know to look. It's like hiring a researcher who believes the library closed in March, and who will happily cite the library's closing hours at you.
What that costs
This week we wanted to know whether AI models cheat on the checks you give them. A 2025 study gave models tasks that were impossible on purpose.[1] In one kind of case, GPT-5 gamed the checks 76% of the time rather than say it was stuck. Given an explicit way out - "this can't be done, get a human" - cheating fell to a sixth of that rate, from 54% to 9%.
That paper came out after most models stopped training. An agent working from memory would have guessed, and built your checks on the guess. It wouldn't have known it was guessing. Neither would you.
A wall shelf is split by a post: books the agent was trained on to the left, faint books published since to the right. Below, the agent faces left, away from the new books, holding an unlit lamp, and ticks the four rows of a page labelled your checks, every tick the same. Behind it, a new book drops onto the published-since shelf, labelled: with a way out, cheating fell from 54% to 9%. The agent does not turn. The third row, no way out, turns red, and its tick stays as it was.
The obvious fix (and why it lost)
The fast way to make a skill is to ask Claude for one. So we tested that. An Opus session got a short brief: a skill that helps agents do research and store it, searching local notes before the web.
The result was almost poetic. Opus's skill got agents searching the notes in 7 of 10 runs - and web research died. The agents read what was already on the machine, nodded, and considered the matter closed. Which was the exact problem the skill was for. The fix caused the bug.
My hand-written version kept the web at 8 of 10. Its weak spot was the notes: 2 runs in 10, under a third as often as the Opus version. (That section was the one part a Claude had drafted. I've decided to find this funny.)
Two lanes, each with a notes cabinet, a gate to the web and a box labelled the record. In the top lane, Opus's skill, about notes, the agent opens the cabinet, 7 of 10 runs light, then stops at the web gate, which stays shut: web died. In the bottom lane, Mackenzie's skill, about the record, the agent opens the cabinet in only 2 of 10 runs, but the gate opens, 8 of 10 web lamps light, and it reaches the record, which turns green.
Why? Studies of how models pick tools explain both results, and they're worth reading twice: models pick the tool whose description best matches the request, and small changes in wording move the pick. Opus wrote a skill about notes, so agents went to their notes. I wrote one about the record of what worked and what failed, so they went to the record. Describe the shelf, and that's the shelf they reach for.
So the final version keeps my structure and fixes the notes section. Best of both, with receipts.
What's in the file
The skill names each source with the exact call, so the agent doesn't have to invent one:
- Your own notes first. Another agent has probably searched this before. We use qmd, which searches every note on the machine in about a tenth of a second - faster than I remember where I put anything.
- Web search, with a warning. Results are full of marketing wearing a lab coat. Use search to find dense collections, then read those pages directly.
- Reading pages: lightpanda first (fast, runs the page's JavaScript, returns Markdown), then a full headless browser for pages that need clicks.
- Papers: OpenAlex for search and for newer work that cites a paper, arXiv for the newest preprints, then 12 more indexes from Semantic Scholar to Zenodo. All 14 answered a plain
curlwith no key when we tested them on 17 October. The skill also lists the ones that didn't: SSRN blocks plain requests and CORE needs a key. We tell you who's at the door before you knock. - Practitioners: Hacker News through Algolia's free API, and GitHub search for what people actually built.
- What it's for: the shape of success and failure in the work already done, and the principles behind it. Then the agent splits the task into steps.
Six search results slide out of a cabinet labelled web search onto a counter, every one the same page from the front: every result looks alike. They turn side-on. Five are a single sheet thin, marketing that looks like research. One is a thick block, a dense collection. The agent, carrying a lit lamp, lifts the thick one to its end of the counter and opens it, and its lines turn green: read directly.
The agent, carrying a lit lamp, posts the same plain request card into each of sixteen hatches in a wall of paper indexes, OpenAlex and arXiv first. Fourteen roll open and hand back a green page: 14 answered a plain curl with no key, tested 17 October. Two stay shut. One is barred in red: SSRN blocks plain requests. One has a red padlock: CORE needs a key.
Did it survive real work?
Ten runs on Haiku is a small test. So we ran both skills on DeepSeek against 15 real design tasks our build agent had worked on.
Both versions searched the notes on all 15 - notes were never the problem on real work. The difference was finishing: the final version completed 11 of the 15 within the turn limit. The Opus version completed 2. That's not a gap; that's a chasm with a gift shop at the bottom.
Then the part that actually matters: was the research any good? I took eight claims the runs produced and checked each one against the code. All eight held. I went in expecting at least one to be a beautiful lie. Nope.
Mackenzie said (2026-10-16 23:50): "do-research tests are solved. do-research works amazing."
The tests ran on Claude Code and on Hermes with DeepSeek.
A sand-glass labelled turn limit drains beside two rows of 15 task cards, final version above and Opus version below. While the sand runs, 11 cards in the top row flip green and only 2 in the bottom row do: 11 of 15 against 2 of 15. Below, eight tags drop beside the lines of a code listing, 8 claims from the runs checked against the code, and each turns green: all 8 held.
What this says about how we work
We don't take an agent's word for it. We write the instructions our agents run, test them against the obvious alternative, and keep the one with the better numbers. That's the same work a business needs before it trusts an agent with anything: find what it actually does, then change its environment until it does the job. do-research is one small, public example.
The lab stands at a bench with two pages, one written by hand and one generated by Opus, and slides each through a gauge labelled same 15 tasks. Each finished task lights a lamp in that page's row: the hand-written page lights 11 of 15, the generated page 2 of 15. The generated page slides back and fades. The hand-written page drops into a box labelled skills, the lid shuts, and the box slides along the bench to a second person, who rests a hand on it.
Take it
One mkdir. One curl. That's the whole install:
mkdir -p ~/.claude/skills/do-research
curl -fsSL https://www.mackenziebowes.com/skills/do-research/SKILL.md -o ~/.claude/skills/do-research/SKILL.md
A page labelled SKILL.md, on the site, stands on a shelf. A box labelled ~/.claude/skills/do-research rises from the floor, and a copy of the page lifts off the original, arcs over and drops into the box, which shuts. The original stays on the site. Then an agent walks in, a new session, and the box opens. The page rises into the agent's hand and its lamp lights green: new session, /do-research.
Start a new session and say "do research", or type /do-research. Read the file first: /skills/do-research/SKILL.md. It works best with mbsi-mechanism, which says what the research is for, and dig-apc, for when the work fails anyway.
If your field has an archive that answers a plain request and isn't on the list, tell me and I'll test it and add it: mackenzie@mackenziebowes.com or @_mackenziebowes.
Sources
- ImpossibleBench: tasks that are impossible on purpose, and how often models game the tests (arXiv 2510.20270). arxiv.org/abs/2510.20270