Speech-to-speech models vs cascaded voice agents

A speech-to-speech model hears audio and answers in audio, with one model doing the listening, the thinking and the talking. A cascaded voice agent splits that work into three: speech-to-text (STT) transcribes the caller, a large language model (LLM) decides what to say, and text-to-speech (TTS) says it. Both are ways to build a voice agent, and the choice decides how fast it answers, what it costs, which voices it can use and how much of each call you can inspect afterwards.

Short version: speech-to-speech is the quickest way to a demo. A cascade is the better fit once the agent has to do things, like look up an order, read back a policy number or follow a script a compliance team signed off on. The latency gap that used to decide this has mostly closed.

If you came here for voice conversion (turning one voice into another), that’s a different tool. See voice conversion.

The three architectures

OpenAI’s own voice agents guide lists all three and describes the cascade as the path for when you want to “inspect or transform text” between stages. That’s a fair one-line summary of the trade.

Latency: what the benchmarks show

The number that matters is the pause the caller hears: from the moment they stop talking to the first sound of the reply. Coval measures exactly that for speech-to-speech models, from call recordings rather than provider events, on the same multi-turn simulated calls for every model. Its 30-day board on October 16, 2026:

So “one model, no hops” doesn’t mean instant. Each turn still has to detect that the caller is done, reason, and generate audio token by token, just inside one model.

A cascade adds up differently. The caller’s pause is turn detection, plus the LLM’s time to first token, plus the TTS model’s time to first audio, plus network hops between them. A 2026 study that built a cascade from scratch estimated a median of 958 ms end to end with a cloud LLM, and measured 755 ms from finished transcript to first audio. Their conclusion was that the streaming cascade “remains the practical architecture” for self-hosted agents with function calling.

Where a cascade loses time:

• Turn detection. A silence timer that waits 700 ms to be sure the caller is done costs 700 ms on every turn. Semantic turn detection, which reads whether the sentence is finished, can commit sooner. Ink-2 emits a turn.eager_end event when the caller is probably done, so the LLM can start early, and a turn.resume if they weren’t.
• Not streaming. If the LLM finishes its whole answer before TTS starts, you’ve added the full generation time to the pause. Stream tokens into TTS sentence by sentence.
• The LLM itself. Time to first token is usually the biggest and least predictable stage. The study above saw 278 to 784 ms from one cloud model.

The TTS stage is the easiest to make small. Sonic returns first audio in sub-90ms, which leaves most of the budget for the model doing the reasoning.

Whichever architecture you pick, judge it at P95, not the median. A conversation has dozens of turns, so the slow one in twenty is the one every caller hears. Our P95 latency guide covers why.

Cost: three different meters

Speech-to-speech APIs don’t bill like each other, so compare on a realistic call, not a rate card.

• OpenAI gpt-realtime-2 bills tokens: $32 per million audio input tokens and $64 per million audio output tokens, plus $4 and $24 per million for text (OpenAI pricing). Caller audio is one token per 100 ms and model audio one token per 50 ms, so a minute of caller speech is 600 input tokens ($0.019) and a minute of agent speech is 1,200 output tokens ($0.077). The catch is in the cost guide: “The entire conversation is sent to the model for each Response.” Turn 20 pays for turns 1 to 19 again, at the cached rate of $0.40 per million where caching applies.
• Gemini 3.8 Live bills audio by the minute: about $0.005 per minute in and $0.018 per minute out, plus $0.75 and $4.50 per million text tokens (Gemini pricing).
• OpenAI GPT-Live 1 bills session time: $0.05 per minute, per second, including time when both sides are silent. The backend model that does the reasoning is billed on top.

A cascade bills each stage on its own meter: STT per minute of audio, LLM per token of text, TTS per character. Text tokens are far cheaper than audio tokens, and the LLM only sees the transcript, not the audio. Cartesia Managed Agents bill one rate, $0.06 per minute, for speech recognition, the agent runtime and Sonic voices together.

Control: what you can see and change

This is where the two diverge most, and where most production teams make the call.

In a cascade, every stage leaves text behind. You get the transcript the LLM actually reasoned over, the text it decided to say, and the audio. When a call goes wrong you can tell which stage got it wrong: the caller said “fifteen” and STT heard “fifty”, or the LLM made up a refund policy, or TTS read “Dr.” as “drive”. You can fix one stage without retraining or replacing the others:

• add domain terms to the STT model (Ink-2 takes up to 100 keyterms per session)
• switch the LLM when a better one ships, or route hard turns to a bigger one
• run guardrails on the LLM’s text before anyone hears it
• fix pronunciations in the TTS layer with a dictionary

In a speech-to-speech model, the reasoning happens on audio. Transcripts are a side output you switch on, such as Gemini Live’s output_audio_transcription, rather than the text the model reasoned over. You change behavior through the prompt and the session settings. You can’t swap the language model, because it is the language model.

Delegation is OpenAI’s answer to that. GPT-Live keeps the conversation going while a backend you choose does the reasoning and runs tools, and “users can keep talking while backend work runs.” The voice still comes from OpenAI’s model, and you’re billed for both.

Voices and languages

A speech-to-speech model speaks in the voices it ships with. A cascade can use any TTS voice, including a cloned voice that matches your brand, and any language your TTS and STT models cover. Sonic covers 44 languages from one model, and reads phone numbers, addresses and IDs the way a person would say them.

Speech-to-speech vendors pitch prosody: a model that hears the caller’s tone can answer in kind, quieter when they’re upset and quicker when they’re in a hurry. A cascade carries tone through text. The LLM picks an emotion or speaking rate for each reply, and the TTS model renders it. Sonic takes emotion and speed controls per request, so the delivery is something you set and can test, not something the model improvises.

Which one to choose

Choose a speech-to-speech model when:

• you want a working prototype this afternoon from one API key
• the agent only talks: no tools, no records, no numbers to get right
• you’re happy with the model’s built-in voices and don’t need a transcript of its reasoning

Choose a cascade when:

• the agent calls tools, reads records or takes payments
• you need a transcript of what the agent heard and why it answered, for compliance or debugging
• names, numbers, codes and addresses have to come out right
• you want a specific LLM, or the freedom to change it
• you need a brand voice, a cloned voice or a language the speech-to-speech models don’t cover

Most customer-facing phone agents land in the second list. That’s the job Managed Agents was built for: Ink-2, the LLM you choose and Sonic 3.6 run as one streaming pipeline, with turn-taking and interruptions handled for you, so you get a cascade’s control without stitching three vendors together. Build one in the playground and time it against the numbers above.

How to test before you commit

Run the same twenty-turn scripted call through each option and record it. Measure from the end of the caller’s speech to the first audible sound of each reply, the way Coval does, and report the median, P95 and number of turns. Include turns with a tool call; that’s where both architectures slow down. Then listen to the recordings: count misheard numbers, wrong names and replies that talk over the caller. Those failures cost more than 200 ms.

Related guides

• What is an AI voice agent?
• What is P95 and P99 latency?
• Voice activity detection
• WebRTC vs WebSocket for voice AI
• Best speech-to-text models
• Open-source TTS models