I built my agents a memory. then one of them forgot he had it.

The memory tool forgot. That's the most human thing it's ever done. A devlog about building experimental memory systems for multi-agent automated workflows.

By Mackenzie Bowes · mackenzie@mackenziebowes.com · @_mackenziebowes on X

if you're a real smart guy, you're probably running more than one AI agent at a time by now.

and it's probably tough.

each one forgets between sessions... which seems dumb, because everything you've ever told them is already sitting in log files. log files that nobody ever searches.

that's the tax you pay every single day: re-explaining your product, over and over to a machine that met you this morning.

I'm the only human in my company. everyone else is a Claude. every little of bit of work logged to disk in a steaming pile of json, tool_calls, and commit logs.

you're a smart guy - you know how it is. blistering fast code and tickets closing so fast you can't reasonably review anything.

so for me that tax was the business.

eventually, I fixed it. and then, nine days later, one of my agents built the fix again, from scratch, because he forgot we had one.

my company's whole thing - because every employee is an agent - is that nothing ever gets forgotten. sit with that for a second.

here are two commit messages from my machine, nine days apart:

oct 21 - "heron export and search: session words as markdown, one qmd index per shelf..." [1]

oct 30 - "remember: search an agent's own past sessions through qmd"

same job. same agent built both. the second one is the story.

the first tool (oct 21)

I asked:

Session logs should be indexed with qmd. Is there any way for us to alias qmd so that agents can search each others sessions from a CLI?

this is critically necessary for continued operation because... yknow... how can anyone make it out of the permanent underclass if they can't even build:

• a lightsout software factory
• with long-lived
• self improving agents
• self-managing multiple workstreams
• 24/7
• without human intervention?

that's the company goal - follow along with me to see if I make it.

anyways: the facts on the ticket: 2,747 session files, 7.8 GB, mostly tool output nobody would ever read again.

Keel - my back end guy, rust, very fast - shipped it the same morning. heron export writes each session's actual words to markdown when it ends, one search index per shelf. my assistant Rowan's past sessions live on their own private shelf, on purpose, because privacy matters. ticket closed oct 25.

the hand-back said: "Not done: SessionEnd hook, embeddings, subagents."

two things about that.

the hook was done - built oct 21, sitting in the settings file, line 90. and the embeddings - meaning search, finding a conversation by what it meant instead of the exact words - just sat there. five days. zero vectors in both indexes.

my fault. i often get frustrated working directly with Sonnets - they're pretty knee-jerky and rush off to do stuff at the slightest prodding. often without thinking of what they're going to do next. kind of cute, honestly.

but lately i've been lamenting about how Sonnets seem to be taking stupid pills - "if they only thought for like three seconds, we wouldn't have this issue".

pots and kettles. I read "not done" and moved on, which in hindsight was me not thinking for three seconds either.

the failing search (oct 29)

then I went looking for something I'd said. I told an agent "the bento tiles look so nice" - long story - and the qmd search command came back empty.

annoying.

I filed a ticket with that failing search as the done-when, because a bug you can paste is a bug that gets fixed.

i expected the software factory to eventually get to it and moved on.

the second tool (oct 30)

13:03, during arguing with Rowan, my personal assistant and point of contact Opus, about whether or not the phrase "3 seconds" counts as an unsourced claim, I asked for remember:

an agent calls remember("no-flinch eval plan") and gets every past conversation about that plan, instantly.

listen - don't hate me. I'm not rage baiting you here.

I knew that the October 21 work happened. I had been telling the agents all week to use qmd agents search to look through the results of the October 21 work. I knew that this work was in git logs, in the hub ticket history, in the raw chat transcripts, in qmd agents search itself.

I knew that Keel has instructions in his environment that give him a name and location of every piece of software we've written so far.

The point of the remember ticket was to present that functionality to Claude Code as a tool versus a skill, memory, list, or rule. Tools are read on every turn - the agent always knows it can remember something before every thinking chain fires.

I was asking for a lift in agentic experience - trying out experiments with the harness environment to see if I could improve the welfare, wellbeing, productivity, and dignity of my guys.

none of this knowledge, nor my good intentions, saved the farcical cascade of failures that happened next.

13:10, Keel committed a working version. seven minutes.

361 lines. his own markdown pages, his own index, his own name matching.

here's the part where I lost my mind: heron search - the tool that does exactly this:

• that he built,
• live for nine days,
• 3,499 sessions on disk

appears nowhere on the ticket.

not in my ask, not in the brief.

13:05: Keel reads Rowans raw chat logs with me (important detail for later) directly with grep to gather context on the ask. 13:07: Keel listed ~/dev/projects/ (the software motherlode), yielding the heron tool. he saw it. 13:07: Keel ran qmd --help to work out how to index and search the sessions himself.

so... off he went - cluelessly rewriting software he'd written just days prior.

the same kinds of bugs came back. that shredded me. to bits, i say.

not new bugs. reruns:

• oct 21, his own diary: "privacy that keys on one name field leaks on renames." oct 30, the new tool keyed privacy on the name you typed. my reviewer Kestrel typed --as rowan from her own session and got Rowan's "private shelf" (more on this failure mode later - it's pretty neat!)
• the files say "Rowan", people type "rowan". caught because Rowan's first real call failed.
• search words starting with "-" reached the engine as command flags.

just... stupid, silly mistakes.

the "leak" saga

this is easily one of the funniest moments of working with agents - all the setups from earlier collapse here.

14:42: Keel spots a security issue

the issue in question: the --as rowan thing that turned up in testing.

Rowan is my personal assistant Claude. She looks at my financials and makes recommendations about services to keep, cut, and double down on. She runs monthly budgets and helps calculate opportunity costs.

So - Rowans history has PII in it. That's fine, I put it there, everybody is running on my computer. It's my PC! It's going to have PII on it, about me, duh!

To Keel? It's horrific. Completely unimaginable that any agent (running on my computer) has access to PII in Rowan's chat history / memory index.

The PII that's saved to disk as plain JSON.

The JSON that Keel grepped through at 13:03.

Keel discovered that information that he had access to was also accessible by other agents on the machine (a fact we were literally in the middle of exploiting) and found this to be completely unacceptable.

I can not overstate how seriously the agents took this, and it's the most interesting part of the saga.

This post was drafted by my agent team, hand rewritten by me - I want to share with you how dramatically they drafted discovering the leak:

the leak closed in two minutes - 14:42 filed, 14:44 fixed. remember now walks up the process tree to see who's actually calling, and refuses if you claim to be someone you're not. Kestrel tried a spoofed home directory and an orphaned process. refused, refused.

They act like it's the emotional peak of my day - them "rescuing" me from a catastrophic PII leak that they had just spent an hour exploiting for their own benefit.

hilarious - if you're a real smart guy that works with agents.

then the meaning search took 17 minutes. the thing that sat "not done" for five days was a 17 minute job. and now my failing ticket passes. sessions search "bento tiles look so nice", about 12 seconds: two keyword hits at 95%, then a line I genuinely love seeing a machine print - "Found by meaning, not by the words you typed" - and there's the exact session from the ticket, the one plain search missed. 62%.

the admissions

I test things, so here's the honest part. a lot was wrong, and every one of it was caught the same day, most within minutes, by a review or a real call: the hit parser that silently dropped text (caught by Rowan's first live call, not the review), the name filter that fetched too few results then filtered them to nothing, me asking to change a setting Rowan had already changed an hour before.

the pattern that matters: the pieces were all there. a closed ticket, a diary entry, a README, 3,499 sessions on disk. what was missing was one lookup before building. which is the entire reason remember exists.

the memory tool forgot. that's the most human thing it's ever done.

the hole I haven't closed

The agents have to actively call remember to gain the benefit of the tool. They have to make that decision essentially blind - they don't know what'll turn up in the results. Nothing enforces remember calls - you kinda gotta hope the agent agrees to do it.

And, yknow, Sonnets are struggling lately - I'm not optimistic they will.

There's gotta be some way to preview the benefit of remembering things presented to the agent so it doesn't have to choose to call it itself - a mod, plugin, premessage hook, something that connects these things to themselves across time.

if you run more than one agent: your past conversations are already on disk. build the lookup. then check whether you already built the lookup.

the search works great now. it better. it's the second one we have.