Holdout scenarios

Checks stored outside the codebase, where the agent cannot see them.

By Mackenzie Bowes · mackenzie@mackenziebowes.com · @_mackenziebowes on X

StrongDM moved its checks to scenarios stored outside the codebase, "similar to a 'holdout' set in model training", judged by an LLM (StrongDM, Software Factories).

Is a

  • Guard: Something that stops a kind of failure before it reaches the end of an unattended run.

Guards