Near-perfect verifier
A check that is close to right, so the agent cannot solve the wrong problem.
Carlini: "it's important that the task verifier is nearly perfect, otherwise Claude will solve the wrong problem". A known-good compiler served as the oracle, which let 16 agents work at once (Anthropic, Building a C compiler).
Is a
- Guard: Something that stops a kind of failure before it reaches the end of an unattended run.
Guards
- Wrong goal: The agent optimises what was stated, not what was meant.