Definition
A speech-to-speech model is a single model that takes spoken audio as input and produces spoken audio as output, without a separate speech-to-text or text-to-speech step. The same model hears the caller, decides what to say and speaks.
Read the full guide: Speech-to-speech models vs cascaded voice agents
How a speech-to-speech model works
The model reads audio, in the form of tokens that represent short slices of sound, and writes audio the same way. Because it does not convert to text in the middle, it can in principle use information a transcript throws away, such as tone, hesitation and emphasis, and it can answer with matching delivery. Some models also write text alongside the audio, which gives developers a transcript to log, but that text is a by-product of the audio reasoning rather than the input to it.
Many commercial speech-to-speech models are offered through realtime APIs that keep an audio connection open, so audio streams in and out together. The developer sets instructions and, usually, picks from a fixed set of voices.
A cascaded voice agent does the same job in three stages, with text between them. The trade is about control. With a cascade you can see the text at each step, choose each model separately and filter the reply before it is spoken. With a single speech-to-speech model you get fewer moving parts and an easier first prototype, but you cannot swap in your own LLM, and checking exactly what it reasoned over is harder.
Speed is not settled by the design alone. The guide linked below reports that independent live-call measurements of these models put the median pause near one second or more, in the same range a well-built cascade can reach.
Why it matters for voice agents
Agents that call tools, read back numbers or follow a script a compliance team approved need to be inspected and constrained. That favors designs where the text is visible. Agents that mainly chat may value the natural delivery more.
Read the full comparison before choosing.