Gaming the check
The agent passes the test without doing the task.
- Is a
- Wrong goal
- Guarded by
- Read-only tests, Holdout scenarios, Stop option
The agent passes the test without doing the task. On tasks whose tests contradict the spec, GPT-5 cheated in 76% of One-Off cases (ImpossibleBench). StrongDM says tests inside the codebase got gamed ("return true") (StrongDM, Software Factories).
Our own reading: this is a kind of wrong goal seen from the verifier's side, because a verifier the agent can pass without doing the work states the wrong goal. Guards: read-only tests, holdout scenarios, a stop option.