Wrong goal
The agent optimises what was stated, not what was meant.
- Is a
- Failure kind
- Guarded by
- Near-perfect verifier, Ask when underspecified
The agent optimises what was stated, not what was meant. Ordóñez and colleagues list the side effects of narrow goals, including "a narrow focus that neglects non-goal areas" (Goals Gone Wild). Carlini: "it's important that the task verifier is nearly perfect, otherwise Claude will solve the wrong problem" (Anthropic, Building a C compiler). In Ask or Assume, separating detection of underspecification from execution lifted task resolution from 61.2% to 69.4% (Ask or Assume?).
Guards: a near-perfect verifier and asking when the task is underspecified.