Holdout scenarios
Checks stored outside the codebase, where the agent cannot see them.
StrongDM moved its checks to scenarios stored outside the codebase, "similar to a 'holdout' set in model training", judged by an LLM (StrongDM, Software Factories).
Is a
- Guard: Something that stops a kind of failure before it reaches the end of an unattended run.
Guards
- Gaming the check: The agent passes the test without doing the task.