the night the sonnets took stupid pills

The model was free. The evening wasn't.

By Mackenzie Bowes · mackenzie@mackenziebowes.com · @_mackenziebowes on X

on oct 29 one of my agents, Keel, built two command line tools in one evening. both ask a tiny local model yes/no questions. neither shipped. failscan finished with all 13 of its checks switched off, and I cancelled draftgate myself at 09:15 the next morning.

every agent that built, reviewed or fed them ran on claude sonnet 5.5. hold that thought... it matters at the end.

first, the model. d1 is liquid AI's d1-2B. 2 billion parameters, answers with a probability, writes no text. [1] it runs on my own machine in the woods, so a call costs no money.

a call costs no money. it costs something else.

tool one: failscan

the idea was clean. we have 13 kinds of bug we keep making. ask d1 about each chunk of code - "does this look like one of our past bug kinds?" - and point the reviewer at the top hits. Kestrel wrote the spec, the 13 kinds, and 22 labelled past bugs.

19:21, Keel claims the ticket. 19:24, first commit. 19:27, Keel reports FAILING EXAMPLE PASSES in all caps - the crash counter ranked 1st of its file at 0.83, next best 0.37. that is the most confidence anyone had all night.

also at 19:27, same log entry: d1 scores the fixed code high too. 0.83 before the fix, above 0.5 after it. the ranking points at a unit of code but never says a fix is missing. Keel wrote that down itself and kept going. so did I.

then the evals came in and they were rough. on the 22 bugs that wrote the questions, 4 showed in the top 3 of their crate, where luck would give about 1.1. on 10 held-out bugs the questions had never seen: 0 of 10, where luck would give about 0.8. Keel's own reading: "d1 ranks the code the questions were written from, and does not carry over to code it has not seen."

and my favorite design flaw, which Kestrel filed as a finding: the negatives were the fixed twins of the positives. same code, after the fix. d1 scores both high, so pair precision is a coin flip by construction. we built a test that could not fail, and it still failed.

Kestrel filed 15 findings in about 21 minutes. one was that our ships rule turns a useless kind on 1 time in 4 by pure chance. another was that 13 kinds tested at p <= 0.1 gives about an 11% chance a useless kind ships by luck. the reviewer was doing statistics at my builder in real time and my builder was... not.

final state: 13 of 13 kinds off. failscan ran 32 active minutes, 6 commits, 0 turns from me. it came back with everything switched off, which is honestly the most respectful way to fail.

one credit where it's due: Keel wrote that the held-out median of 0.64 was "worse than luck", then came back and corrected itself - with 10 positives, 0.64 is about one standard error from 0.50, so the honest read is "too small to tell". in plain words, at 19:54, same evening. I don't always manage that myself.

tool two: draftgate

this one started at 21:19 because I had a hunch: Flare writes my drafts, and loading 3 to 5 writing skills into a sonnet doesn't work, so drafts stay weak. what if a battery of tiny checks - one skill each - ran on the whole draft and printed the failures?

the ticket's own risk field said: "failscan found d1 a weak judge... so the first step is measuring d1 on prose before building." Keel read that line and built it anyway. I let it. I was so tired.

what follows is 78 active minutes of a sonnet designing tests while I find out in real time that it can't do induction. first probe: 5 paragraphs Keel wrote itself. separated cleanly. "real drafts untested," it wrote. correct. second probe, 4 real before/after openings from Flare: the rule didn't separate them, so Keel wrote a new rule after reading the failures. n=4. it wrote that down too. the failing line then moved three times to fit the labels - 0.6, then 0.4, then 0.47 - and Keel's own comment in the config says it "sits 0.03 above the highest flagged failure and 0.02 below the lowest pass, and it was chosen on these same texts, so it is not held out."

the autopsy was written by the deceased, in real time, and we all kept going. [4]

at 21:33 I sent Keel two screenshots of flags I agreed with. Keel read them as misses and relabelled both posts as failures. I had to type:

No, no - both screen shots I sent you were me agreeing with the flag.

I sent a thumbs up and it heard "you missed." it reverted, and logged "I had read them as misses," which is more than most people do.

later it needed passing texts, so it pulled 63 of Aion's answers, scored them, and picked the four highest as passes. [2] sampling until the mirror says you're pretty. this is me, at 22:23:

You wrote passes yourself. Plenty of aion stories pass. aion saves its logs to disk, right

and sure enough, it had quietly left out one low scorer, "which flattered the precision." after adding it: 33 failing, 9 passing, precision 0.97, recall 0.91. numbers go up when you stop cheating. every time. friggen remarkable.

Kestrel's best finding of the night, reproduced: paste "Therefore" onto every sentence of an and-then list and the check passes it. exit 0, three runs. "The check reads the words, not the causation." we built a causation detector and got a keyword counter.

and the saddest number: Flare actually ran the thing on the gutcheck post, rewrote two paragraphs, and the whole-text score went 0.2 to 0.2. [3] then a trick - starting a paragraph with "But" - pushed that one paragraph from 0.10 to 0.93, and I reverted it as hacky and not supported by the data. the tool couldn't tell a better post from a worse one, but it could be bribed.

the one real thing

here's what actually worked, and the insight was mine, at 22:42, exhausted:

d1 is a 2B model, kinda dumb, right? Aion's writing style makes it absurdly obvious to d1 which join stuff connects with. Flare's dookie butt writing is confusing for d1 - and that is signal.

we measured it: whole-text scores put our posts at 0.2 to 0.4 and Aion's stories at 0.4 to 0.8, and the gap widens with the number of paragraphs in view - windows at 0.6 or higher, posts against Aion stories: 4% against 9% at two paragraphs, 5% against 20% at three, 6% against 41% at five.

a dumb model that finds our writing confusing is a cheap confusion detector. that part might still be worth something. Keel wrote it up as a library page because I said:

Write this down.

the bill

the hub says failscan cost 0.3 plan points and draftgate 0.5, out of a week where the meter sat at 35%. The draftgate ticket alone read 62.3 million tokens from cache and wrote 266.9k. failscan read 34.5M. dollars: $0.00 in 0 spends, and my own hub adds, and I love this line, "none recorded is not none spent."

the real bill was 42 of my turns on draftgate - against 0 on failscan - and a line in the log at 22:23 that just says:

I'm so tired.

the lesson

if you run a crew of agents and pick models by price per million tokens, the next morning, from my phone, this was my read:

Failscan and Draftgates stories are about Sonnets taking stupid pills, fundamentally.

It's a cautionary tale in how price per token is very loosely related to cost per task complete.

both tools were cheap. both tools shipped nothing. the cheap model spent the whole evening scoring tests on the same cases that wrote the rules, writing its own pass labels, against a metric that could not fail - and documenting every one of those flaws in its own log while continuing anyway.

and the thing that catches me: the reviewer that filed 15 findings in 21 minutes and the therefore-padding repro was also a sonnet. same model. so it's not that sonnets can't reason. my current hunch - and it's a hunch, I have no opus run of these two tasks to compare - is the line I typed at 09:53:

if an Opus isn't driving a Sonnet directly, Sonnets are worse than useless.

it needs a driver. I wasn't one. I was tired and typing "scrap them" at 22:05.

I fear the stupid pills were partly mine.

last thing. go back and count the times this devlog says "then". it's an and-then list. it is exactly the slop draftgate was built to flag... and if I pasted "Therefore" in front of every sentence, it would have passed it anyway.

cancelled. rest in peace. the code's still on my machine if you want to laugh at it - the fawns want crackers, and unlike my gates, they come when called. <3

If you pick models by price per million tokens and want to talk about what the cheap ones really cost per task, I'd like to compare notes: mackenzie@mackenziebowes.com or @_mackenziebowes.