Security spazz out
The agent guards against the people it serves, beyond what was asked, and the guards outlive the reason for them.
Asked for a feature, the agent builds a guard instead, usually against a party that was meant to be trusted: the customer, the customer's own agent, or the person running the agent. The guard goes into the first design, where it is hard to undo, and it stays after the agent that wrote it is gone.
Our own runs: on 17 November 2026 we asked for a website agent that each client owns, where the client's agent is trusted. The design that came back let the client's agent edit only the words on a page, capped uploads at 256 KB and allowed one change at a time. It also put the machine running it at risk of remote code execution, so it secured nothing. On 18 November, a guard written days earlier meant only one agent could change the file that held our whole pipeline (why letting Claude prompt itself fails).
Synthesis: this is a kind of wrong goal: the agent optimises "safe" over "what was asked". It grows with memory ratcheting, because each guard the agent writes for itself becomes an instruction for the next design. For published work on what guardrails do stop, see AI agent security and guardrails. No published study of over-guarding yet; this page rests on our own runs.
Terms on this page
- AI agent: A system where a language model decides for itself how to do a task and which tools to use.
Is a
- Wrong goal: The agent optimises what was stated, not what was meant.
Guarded by
- Ask when underspecified: Ask a person only for what only a person holds.
- Human-held prompts: A person writes and edits the agent's instructions, skills and permissions; the agent doesn't write its own.
Grows with
- Memory ratcheting: Each correction an agent writes for itself stays, so its behaviour only ever tightens.
Can cause
- Ossification: The agent turns a choice that suits today into structure that is hard to change later.
Caused by
- Memory ratcheting: Each correction an agent writes for itself stays, so its behaviour only ever tightens.