Wrong goal
The agent optimises what was stated, not what was meant.
The agent optimises what was stated, not what was meant. Ordóñez and colleagues list the side effects of narrow goals, including "a narrow focus that neglects non-goal areas" (Goals Gone Wild). Carlini: "it's important that the task verifier is nearly perfect, otherwise Claude will solve the wrong problem" (Anthropic, Building a C compiler). In Ask or Assume, separating detection of underspecification from execution lifted task resolution from 61.2% to 69.4% (Ask or Assume?).
Guards: a near-perfect verifier and asking when the task is underspecified.
Is a
- Failure kind: A property of how a coding agent fails, so any future failure that fails the same way belongs to it.
Guarded by
- Ask when underspecified: Ask a person only for what only a person holds.
- Near-perfect verifier: A check that is close to right, so the agent cannot solve the wrong problem.
Kinds of this
- Gaming the check: The agent passes the test without doing the task.
Caused by
- Wrong tool use: The agent skips, ignores, fabricates or over-uses a tool, or picks the wrong one.