An AI voice agent picks up the phone, hears what the caller wants, and does something about it: books the appointment, checks the order, answers the question, or hands off to a person before the caller gets stuck. Chatbots do this job by reading and typing. A voice agent does it over live audio, while the caller is still mid-sentence.
That timing difference is most of the engineering. This page explains what a voice agent is, how the pipeline works, what makes them hard to build, and the two ways to build one on Cartesia.
What is an AI voice agent?
A voice agent is software that holds a spoken conversation and takes action. It transcribes what the caller says, decides what to do, calls tools such as a calendar or an order system, and speaks the result back. “Conversational AI” is the broader category it belongs to; a voice agent is that category applied to real-time speech. We keep a separate conversational AI overview for the category as a whole.
Two neighbors get confused with it:
| System | What it does | Why it’s different |
|---|---|---|
| Touch-tone or menu IVR | Plays menus, routes calls | No language understanding; the caller works through the menu, not the other way around |
| Text chatbot | Reads and writes text | No live audio, and it can pause between turns without the caller noticing |
A voice agent can understand a request the menus never anticipated, and it must reply while the caller is still on the line. Miss the timing and callers talk over it, wait through awkward gaps, or hang up.
How a voice agent works
The loop has four stages, and the good ones run concurrently rather than one after another:
| Stage | Job | What to watch |
|---|---|---|
| Streaming speech-to-text | Transcribe the caller’s audio as it arrives | Accuracy on your accents, phone codecs, and noise; don’t drop silence the turn detector needs |
| Turn detection | Decide when the caller has finished their thought | A pause says “thinking” or “done”; a fixed timeout treats both the same |
| LLM and tools | Plan the reply, call the calendar or order system | Tool latency lands straight in the caller’s wait time |
| Text-to-speech | Speak the reply, streaming as it generates | Start speaking before the full sentence exists; handle barge-in cleanly |
The caller experiences one number: the time between finishing their sentence and hearing the reply. Every stage contributes to it. A sub-second text-to-speech model doesn’t rescue an agent whose tool calls take three seconds, and a fast LLM doesn’t rescue a recognizer that garbles the caller’s account number. When you measure, measure the whole loop. Cartesia’s Ink handles the first two stages in one model: streaming transcription with native turn detection. Sonic is the fourth stage, a text-to-speech model ranked #1 in blind listening tests, with sub-90ms model latency and 40+ languages. The language model in the middle is yours to choose.
What makes voice agents hard
The pipeline is the easy part. Three behaviors are where agents fall apart:
Turn-taking. People don’t speak in finished sentences. They say “my address is…” and stop to look it up. Cross-linguistic studies of conversation find people hand the floor to each other in about 200 milliseconds (Stivers et al., 2009), so an agent that waits a full second to be sure has already lost the rhythm, while one that jumps in on a pause interrupts. Deciding whether a pause means “thinking” or “done” is its own problem; we wrote a guide to voice activity detection and turn detection that covers it.
Interruptions. Callers change their mind mid-reply: “actually, make that Thursday.” A useful agent stops talking, absorbs the correction, and continues. That means canceling not just text generation but the audio already buffered in the caller’s speaker, which is a client-side problem as much as a server-side one.
Recovery. The recognizer will mishear. The tool call will time out. The agent needs a next move: ask once to confirm, retry, or transfer to a human before the caller repeats themselves for the third time. Agents that can only succeed are agents that fail loudly.
Evaluate against your calls, not a demo
Every vendor’s demo works, because the demo was written for the demo. Your callers have your accents, your phone connection, and your terminology. Before routing live traffic, run a batch of recorded or test calls through the agent and score the conversation:
- Does it let callers finish, including trailing thoughts like “and, um, the second one”?
- Does it start replying promptly after complete answers, without a dead second?
- When the caller interrupts, does the audio actually stop?
- When it mishears, does it confirm or transfer instead of guessing?
- When the action is consequential (cancel a booking, change an order), does it confirm before doing it?
The AI call center pilot guide turns this into a scorecard for one support queue, and the AI receptionist guide includes an acceptance-test table with a prompt-injection case.
Two ways to build one on Cartesia
Managed Agents (/agents) wires the loop for you. Pick your LLM, connect tools and transfers, add a knowledge base, and get a phone number, in a few clicks. Cartesia runs the streaming stack underneath. It’s the fast path from “nothing” to “testable agent”.
The API gives you the pieces. Sonic for speech, Ink for streaming transcription and turn detection, your own LLM in the middle, and your code owns the loop. Use it when you need control over the model, the host language, or the deployment. The Managed Agents introduction in the docs covers both paths and what each one manages for you.
Both paths sit on the same models. They’re built on State Space Model architecture, which is what makes streaming at these latencies practical, but you don’t need to care about that until you care about your bill.
Where voice agents are used
Customer support and help desks, receptionists and scheduling, call-center automation, and interactive experiences such as characters and toys. Decagon, ServiceNow, Quora, Retell, and EliseAI build voice products with Cartesia; the customers page has the specifics.