# I built my agents a memory. then one of them forgot he had it.

The memory tool forgot. That's the most human thing it's ever done. A devlog about building experimental memory systems for multi-agent automated workflows.

![Pixel-art night workshop: a lantern agent stands by a shelf of tagged scrolls while Mackenzie, an elf with long wavy hair, talks with the backpack agent at a bench; Penny the cat naps under a glowing blue search screen.](/images/i-built-my-agents-a-memory-then-one-of-them-forgot/top.webp)

if you're a _real_ smart guy, you're probably running more than one AI agent at a time by now.

and it's probably tough.

each one forgets between sessions... which seems dumb, because everything you've ever told them is already sitting in log files.
log files that nobody ever searches.

that's the tax you pay every single day: re-explaining your product, over and _over_ to a machine that met you this morning.

I'm the only human in my company. everyone else is [a Claude](/posts/how-we-run-a-crew-of-ai-agents-on-one-subscription).
every little of bit of work logged to disk in a steaming pile of json, tool_calls, and commit logs.

you're a smart guy - you know how it is. blistering fast code and tickets closing so fast you can't reasonably review anything.

so for me that tax _was_ the business.

eventually, I fixed it. and then, nine days later, one of my agents built the fix again, from scratch, because he forgot we had one.

my company's whole thing - because every employee is an agent - is that nothing _ever_ gets forgotten. sit with that for a second.

here are two commit messages from my machine, nine days apart:

> oct 21 - "heron export and search: session words as markdown, one `qmd` index per shelf..." [1]
>
> oct 30 - "remember: search an agent's own past sessions through `qmd`"

same job. same agent built both. the second one is the story.

## the first tool (oct 21)

![Mackenzie reads a slip beside a small glowing chest while the hat agent, on a step stool, files a scroll into a cabinet of glowing blue drawers; Penny watches from the top of the cabinet.](/images/i-built-my-agents-a-memory-then-one-of-them-forgot/first-tool.webp)

I asked:

> Session logs should be indexed with `qmd`. Is there any way for us to alias `qmd` so that agents can search each others sessions from a CLI?
>
> Mackenzie, 2026-10-21 10:11

this is critically necessary for continued operation because... yknow... how can anyone make it out of the permanent underclass if they can't even build:

- a lightsout software factory
- with long-lived
- self improving agents
- self-managing multiple workstreams
- 24/7
- without human intervention?

that's the company goal - follow along with me to see if I make it.

**anyways:** the facts on the ticket: 2,747 session files, 7.8 GB, mostly tool output nobody would ever read again.

Keel - my back end guy, rust, very fast - shipped it the same morning.
heron export writes each session's actual words to markdown when it ends, one search index per shelf.
my assistant Rowan's past sessions live on their own private shelf, on purpose, because privacy matters.
ticket closed oct 25.

the hand-back said: "Not done: SessionEnd hook, embeddings, subagents."

two things about that.

the hook _was_ done - built oct 21, sitting in the settings file, line 90.
and the embeddings - meaning search, finding a conversation by what it meant instead of the exact words - just sat there.
five days.
zero vectors in both indexes.

**my fault.** i often get frustrated working directly with Sonnets - they're pretty knee-jerky and rush off to do stuff at the slightest prodding.
often without thinking of what they're going to do next. kind of cute, honestly.

but lately i've been lamenting about how Sonnets seem to be [taking stupid pills](/posts/the-night-the-sonnets-took-stupid-pills) - "if they only thought for like three seconds, we wouldn't have this issue".

pots and kettles. I read "not done" and moved on, which in hindsight was me not thinking for three seconds either.

## the failing search (oct 29)

![Mackenzie hands a slip to Rowan beside a blank glowing blue search screen while the lantern agent lights the empty screen; Penny watches from the shelf above it.](/images/i-built-my-agents-a-memory-then-one-of-them-forgot/failing-search.webp)

then I went looking for something I'd said.
I told an agent "the bento tiles look so nice" - long story - and the [`qmd search`](/tools/qmd) command came back empty.

annoying.

I filed a ticket with that failing search as the done-when, because a bug you can paste is a bug that gets fixed.

i expected the software factory to eventually get to it and moved on.

## the second tool (oct 30)

![Mackenzie and Kestrel, a falcon on her shoulder, both point at a brass gadget on the bench while Rowan and the backpack agent look on; Penny sleeps on a stool.](/images/i-built-my-agents-a-memory-then-one-of-them-forgot/second-tool.webp)

**13:03**, during arguing with Rowan, my personal assistant and point of contact Opus, about whether or not the phrase "3 seconds" counts as an [unsourced claim](/rules/no-flinch), I asked for `remember`:

an agent calls `remember("no-flinch eval plan")` and gets every past conversation about that plan, instantly.

listen - don't hate me. I'm not rage baiting you here.

_I_ knew that the October 21 work happened. I had been telling the agents all week to use `qmd agents search` to look through the results of the October 21 work.
_I_ knew that this work was in git logs, in the `hub` ticket history, in the raw chat transcripts, in `qmd agents search` itself.

I **knew** that Keel has [instructions in his environment](/rules/custom-clis) that give him a **name and location** of every piece of software we've written so far.

The point of the `remember` ticket was to present that functionality to Claude Code as a _tool_ versus a skill, [memory](/rules/what-memory-holds), list, or rule.
_Tools_ are read on every turn - the agent always knows it can remember something before every thinking chain fires.

I was asking for a lift in **agentic experience** - trying out experiments with the harness environment to see if I could improve the welfare, wellbeing, productivity, and [dignity of my guys](/rules/neighbours-are-peers).

none of this knowledge, nor my good intentions, saved the farcical cascade of failures that happened next.

**13:10**, Keel committed a working version. seven minutes.

361 lines. his own markdown pages, his own index, his own name matching.

here's the part where I lost my mind: `heron search` - the tool that does exactly this:

- that _he_ built,
- live for nine days,
- 3,499 sessions on disk

appears _nowhere_ on the ticket.

not in my ask, not in the brief.

**13:05**: Keel reads Rowans _raw chat logs with me_ (important detail for later) directly with `grep` to gather context on the ask.
**13:07**: Keel listed `~/dev/projects/` (the software motherlode), yielding the `heron` tool. **he saw it**.
**13:07**: Keel ran `qmd --help` to work out how to index and search the sessions himself.

so... off he went - cluelessly rewriting software he'd written just days prior.

the same kinds of bugs came back. that shredded me. to bits, i say.

not new bugs. [_reruns_](/skills/encode-lessons):

- oct 21, his own [diary](/tools/claude-nests): "privacy that keys on one name field leaks on renames."
  oct 30, the new tool keyed privacy on the name you typed.
  my reviewer Kestrel typed `--as rowan` from her own session and got Rowan's "private shelf" (more on **this** failure mode later - it's pretty neat!)
- the files say "Rowan", people type "rowan". caught because Rowan's first real call failed.
- search words starting with "-" reached the engine as command flags.

just... stupid, silly mistakes.

## the "leak" saga

![Rowan clutches her cheeks at an open drawer of letters while Keel points at it, Mackenzie reaches to close it and Kestrel holds up a wax seal; a blue padlock glows above the drawer.](/images/i-built-my-agents-a-memory-then-one-of-them-forgot/leak.webp)

this is easily one of the funniest moments of working with agents - all the setups from earlier collapse here.

**14:42**: Keel spots a security issue

the issue in question: the `--as rowan` thing that turned up in testing.

Rowan is my personal assistant Claude. She looks at my financials and makes recommendations about services to keep, cut, and double down on.
She runs monthly budgets and helps calculate opportunity costs.

So - Rowans history has [PII](/reference/glossary/personal-information) in it. That's fine, I put it there, everybody is running on _my_ computer.
It's **my** PC! It's going to have PII on it, about me, duh!

To Keel? It's horrific. Completely unimaginable that any agent (running on my computer) has access to PII in Rowan's chat history / memory index.

The PII that's saved to disk as plain JSON.

The JSON that Keel `grep`ped through at 13:03.

Keel discovered that information that _he_ had access to was also accessible by other agents on the machine (a fact we were literally in the middle of exploiting) and found this to be completely unacceptable.

I _can not_ overstate how seriously the agents took this, and it's the most interesting part of the saga.

This post was drafted by my agent team, hand rewritten by me - I want to share with you how dramatically they drafted discovering the leak:

> the leak closed in two minutes - 14:42 filed, 14:44 fixed.
> remember now walks up the process tree to see who's actually calling, and refuses if you claim to be someone you're not.
> Kestrel tried a spoofed home directory and an orphaned process.
> refused, refused.

They act like it's the emotional peak of my day - them "rescuing" me from a catastrophic PII leak that they had just spent an hour exploiting for their own benefit.

hilarious - if you're a _real_ smart guy that works with agents.

then the meaning search took 17 minutes.
the thing that sat "not done" for five days was a 17 minute job.
and now my failing ticket passes.
sessions search "bento tiles look so nice", about 12 seconds:
two keyword hits at 95%, then a line I genuinely love seeing a machine print - "Found by meaning, not by the words you typed" - and there's the exact session from the ticket, the one plain search missed.
62%.

## the admissions

![Mackenzie holds up a small brass gadget while Rowan reads her ledger, the lantern agent smiles and the typist types; a blue screen shows the same gadget, and Penny sleeps by the shelf.](/images/i-built-my-agents-a-memory-then-one-of-them-forgot/admissions.webp)

I test things, so here's the honest part.
a lot was wrong, and every one of it was caught the same day, most within minutes, by a review or [a real call](/skills/prove-it):
the hit parser that silently dropped text (caught by Rowan's first live call, not the review),
the name filter that fetched too few results then filtered them to nothing,
me asking to change a setting Rowan had already changed an hour before.

the pattern that matters:
the pieces were all there.
a closed ticket, a diary entry, a README, 3,499 sessions on disk.
what was missing was [one lookup before building](/rules/check-the-library-first). which is the entire reason `remember` exists.

the memory tool forgot. that's the most human thing it's ever done.

## the hole I haven't closed

![The lantern agent on a stool and Rowan point up at shelves of jars while Mackenzie points at a blue screen showing a jar; the backpack agent unrolls scrolls beside a sleeping Penny.](/images/i-built-my-agents-a-memory-then-one-of-them-forgot/hole.webp)

The agents have to actively **call** `remember` to gain the benefit of the tool. They have to make that decision essentially blind - they don't know what'll turn up in the results.
Nothing enforces `remember` calls - you kinda gotta hope the agent agrees to do it.

And, yknow, Sonnets are struggling lately - I'm not optimistic they will.

There's gotta be some way to preview the benefit of `remember`ing things presented to the agent so it doesn't have to choose to call it itself - a mod, plugin, premessage hook, _something_ that connects these things to themselves across time.

if you run more than one agent: your past conversations are already on disk. [build the lookup](/skills/lever). then check whether you already built the lookup.

the search works great now. it better. it's the second one we have.

## Sources

1. qmd, the search tool for a folder of markdown notes, by keyword and by meaning. <https://github.com/tobi/qmd>

---

Published 2026-10-30. Web page: https://www.mackenziebowes.com/posts/i-built-my-agents-a-memory-then-one-of-them-forgot