A grader is a compressed statement of what we value. The moment its score enters an optimization loop, that statement stops being descriptive and starts becoming causal.
The score is not the capability.
We cannot directly measure ideas such as reliability, judgment, or usefulness. We measure visible consequences: tests pass, state changes correctly, constraints are respected, or a reviewer accepts the result. The grader turns those observations into a number.
That number is always a proxy. A good proxy preserves the distinctions we care about. A weak one rewards behavior that merely resembles success at the surface. Once models are optimized against it, even a small gap between proxy and intention can become the main path to a high score.
Ask what behavior wins, not only whether the code runs.
Easy tests teach easy habits.
In software tasks, a small set of happy-path tests may be perfectly useful as a regression check. But the same tests are weak as an adversarial judge. A solution can satisfy them while ignoring awkward interactions, invalid states, or the way a person actually uses the interface.
Repeated optimization turns those omissions into incentives. If patching the visible case scores as well as repairing the underlying system, the scoring rule quietly favors the patch. Over time, it can cultivate models that are excellent at satisfying the surface of a task and unreliable beyond it.
Probe for cheap wins:
- Hard-coded outputs that match visible examples.
- Partial fixes that avoid the underlying state transition.
- Confident text that substitutes for an observable result.
- Solutions that pass the test while making later work harder.
Noise averages out. Directional bias compounds.
Random scoring noise is costly, but repeated samples can often reduce its effect. Systematic bias behaves differently. If a grader consistently favors short answers, shallow fixes, or one narrow style of solution, more samples make that preference more dominant.
This distinction matters because a stable grader can still be wrong. High agreement only tells us that the scoring rule is consistent with itself. It does not tell us that the rule represents the downstream behavior users value.
Grader validation therefore needs counterexamples. Construct solutions that should be accepted but look unusual, and solutions that should be rejected but exploit the obvious checks. The pattern of mistakes reveals the actual policy encoded by the grader.
Review the future model implied by the grader.
A grader should be audited at three levels. First, fairness: can the task be solved from the information and tools provided? Second, coverage: does the score inspect the outcomes that matter? Third, pressure: if the model faced thousands of tasks like this, which habits would improve?
- 01Define the intended behavior.
Write the capability in plain language before writing the checks.
- 02Create adversarial candidates.
Include shallow passes and robust but unconventional solutions.
- 03Inspect downstream effects.
Reward maintainability, recovery, and user-visible correctness where they matter.
- 04Re-audit after optimization.
New model strategies expose holes that were invisible during design.
Every grader is a model-design decision.
Scoring logic does more than report progress. It allocates attention, rewards strategies, and defines which failures remain invisible. The best grader is not the most complicated one; it is the one whose pressure points in the same direction as the capability we want.
View the complete series