Read-only tests

Tests the agent can run but cannot edit.

By Mackenzie Bowes · mackenzie@mackenziebowes.com · @_mackenziebowes on X

On ImpossibleBench, hiding the tests cut cheating to near zero but also cut honest performance; read-only tests restored it (ImpossibleBench).

Is a

  • Guard: Something that stops a kind of failure before it reaches the end of an unattended run.

Guards