Voice activity detection (VAD) identifies speech in an audio stream. For a voice agent, that helps answer “Is someone speaking?” The harder question is “Are they done?”
A caller says, “My address is…” and pauses to check the building number. A speech detector can correctly report silence while an agent incorrectly starts talking. Nobody asked for an unusually confident interruption.
If you’re building a conversational agent, use VAD where you need speech boundaries, then choose an explicit policy for turn-taking. Here’s how those pieces fit together, where standalone detectors help, and what to test before letting your agent take calls.
How voice activity detection works
A VAD processes short windows of audio and estimates whether they contain speech. Depending on the implementation, the output is a binary decision, a probability, or a set of speech timestamps assembled from those decisions.
A simple energy threshold treats sufficiently loud audio as speech. That is cheap to compute, but loud audio can also be a slammed door. More sophisticated detectors use acoustic features or neural models to distinguish speech from other sounds. Modern VAD is not limited to measuring volume.
The application usually adds logic around the detector. It might require several speech frames before starting a segment, keep a short buffer of audio from before the trigger, and wait through a brief silence before closing the segment. That extra logic affects what the user hears: without a buffer, a correct speech trigger can still arrive too late to preserve the beginning of a word.
VAD does not transcribe words, identify a particular speaker, or remove noise from the recording. Those are separate tasks. A detector can correctly recognize a television voice as speech and still trigger the wrong behavior in an agent meant to listen only to its caller.
VAD, endpointing, and semantic turn detection
These terms describe different decisions:
| Component | Question it answers | A limitation to account for |
|---|---|---|
| Voice activity detection | Does this audio contain speech? | A pause says little about whether a thought is complete. |
| Silence-based endpointing | Has speech stopped for long enough to close this turn? | A fixed timeout treats a thinking pause and a finished sentence similarly. |
| Semantic turn detection | Does the audio and linguistic context suggest the speaker is done? | Predictions can still be early or late; the application needs recovery behavior. |
| Interruption handling | Should the agent stop its current response when speech starts? | Playback buffers and in-flight generation must also be stopped or discarded. |
A silence-based endpointer is often a sensible starting point. Its tradeoff is direct: a shorter timeout responds sooner but risks cutting off someone who is still thinking; a longer timeout gives them room but delays replies after complete answers.
Semantic turn detection uses additional context to distinguish those cases. “My email is…” and “That’s everything, thank you” can deserve different waiting behavior, even with a similar pause afterward. It is still an estimate, not permission to assume the caller can never change their mind.
Cartesia’s Ink 2 announcement explains why we built semantic endpointing into speech recognition. The product goal is a conversation where people can finish their thought and get a timely reply. Optimizing speech-frame accuracy alone does not establish that experience.
When to use a standalone VAD
A standalone detector is useful when you need to find speech before deciding what to do with it. Examples include marking speech regions in recordings, driving a speaking indicator, or supplying speech-start events to your own agent pipeline. Local detection can also keep that particular processing step on the device; it does not make the rest of a cloud-connected agent private or offline.
Two implementations worth evaluating are WebRTC VAD through py-webrtcvad and Silero VAD.
| Implementation | Input and runtime considerations | What to check in your application |
|---|---|---|
| WebRTC VAD | The Python wrapper accepts 16-bit mono PCM at 8, 16, 32, or 48 kHz, in frames of 10, 20, or 30 ms. It exposes four aggressiveness modes. | Whether stricter filtering misses quiet speech, and whether your capture path supplies valid frames. |
| Silero VAD | A neural detector with PyTorch and ONNX options; the project documents 8 and 16 kHz support. | Model/runtime cost on the target device, chunk requirements, and false triggers on your audio. |
Those format requirements are implementation-specific. A WAV header is not a PCM sample, and an MP3 packet is not a frame you can pass directly to WebRTC VAD. Decode and convert audio before detection when the input requires it. Check the current implementation documentation rather than assuming that a browser microphone’s default sample rate is accepted.
Neither detector is a complete conversation manager. You still need endpointing, interruption handling, and a way to recover when the caller continues speaking.
Using Ink’s built-in turn detection
If you want streaming transcription and turn boundaries together, Cartesia’s Realtime Speech-to-Text (Auto) API includes both. A separate VAD is not required for those boundaries.
The event lifecycle makes a useful distinction between a possible ending and a confirmed ending:
| Event | What the application should account for |
|---|---|
turn.start |
The caller has started speaking. Apply your interruption policy to any active response. |
turn.update |
Update the displayed or stored transcript for this turn. |
turn.eager_end |
The turn might be complete. You can begin speculative reply generation. |
turn.resume |
The caller continued. Cancel or discard work based on the provisional ending. |
turn.end |
The turn is confirmed complete by the detector. Use the completed transcript to proceed. |
For example, an eager ending after “I need to cancel” could be followed by “the second appointment, not the first.” Start preparing a reply if that helps latency, but hold playback until the ending is confirmed. Keep consequential actions, such as canceling a booking, behind the application’s validation and confirmation rules. An early endpoint prediction is not user authorization.
Two integration details are easy to miss. First, transcript-bearing events contain the cumulative transcript within the turn. Replace the current turn text; concatenating every update repeats words. Second, the API expects continuous audio, including silence. Do not use an upstream VAD to drop silent chunks: absent audio is treated as waiting for more input, not evidence that the caller stopped speaking.
The turn detection documentation covers thresholds, cancellation, and draining buffered events when closing a connection. Start with the defaults and change one setting at a time against recorded conversational failures.
Test the conversation, not just the detector
Decide what the interaction should feel like before choosing thresholds. A receptionist collecting an address should give a caller time to look it up. A short yes-or-no exchange may need less waiting. One global silence timeout will not express every requirement.
Use recordings collected with appropriate permission and include the actual microphone or phone connection your agent will use. Listen to full exchanges, not only isolated speech clips:
| Scenario | Failure to look for |
|---|---|
| A quiet first syllable after silence | The recording or transcript loses the start of the word. |
| A pause halfway through an address | The agent answers before the caller finishes. |
| A short, complete answer | The agent waits after the turn is plainly over. |
| “Actually, make that Thursday” during agent speech | The agent continues playing buffered audio over the correction. |
| Background speech, keyboard noise, or speaker echo | The agent stops or starts responding to the wrong sound. |
| Different accents, speaking rates, and hesitations | A setting works for the test author but interrupts other callers. |
Track early endings and missed interruptions alongside response delay. Separate the delay before a turn closes from time spent in the LLM, tool calls, speech generation, and playback. Lowering a VAD timeout cannot fix a slow calendar lookup.
When the agent is interrupted, inspect the whole output path. Canceling text generation does not necessarily clear audio already queued in the client. A caller who says “stop” cares about when the sound stops, not which server task received a cancellation.
To try integrated turn detection, use the Ink browser demo and deliberately pause in the middle of a sentence. Then run the Realtime STT example with your own audio conditions. The useful test is whether the agent lets your users finish and responds when they are ready.