Some buyers cannot use a hosted voice API at any price. Government agencies, hospitals, banks, and defense contractors buy voice AI with a condition attached: the audio stays inside our perimeter. On-premise voice AI is how that condition gets met, and it changes what you should evaluate before you buy.
This page explains what on-premise voice AI is, what actually has to run on your hardware, where it sits on the spectrum from cloud API to air gap, and how to evaluate vendors. It ends with how Cartesia does it, including the fully air-gapped on-prem deployment behind ServiceNow’s AI Voice Agents.
What is on-premise voice AI?
On-premise voice AI is speech software that runs on infrastructure your organization owns and controls. The vendor ships the models; your team runs them, on your servers, in your data center or a cloud account you hold the keys to. Nothing about a live call, its audio, or its transcript has to pass through the vendor’s infrastructure.
The contrast is the default deployment most people know: a cloud API where you stream audio to the vendor, the vendor runs the models, and audio and transcripts live under the vendor’s terms of service. That trade is fine for most products and wrong for some. If the audio is a classified briefing, a therapy session, or a call a regulator can demand copies of, “the vendor says they delete it” is not always an acceptable answer.
“Voice AI” here means a real-time speech pipeline, and a voice agent built on one has four moving parts:
| Part | What it does | Can it run on-prem? |
|---|---|---|
| Streaming speech-to-text | Transcribes while the caller is still speaking | Yes |
| Turn detection | Decides when the caller has finished | Yes |
| Language model + tools | Works out the response and calls your systems | Yes (yours to choose) |
| Streaming text-to-speech | Speaks the answer back while it is still being generated | Yes |
Every one of these can run inside your perimeter. The language model was always yours to pick; the parts vendors usually keep to themselves are the speech models, and whether those can come indoors is a real differentiator between vendors.
Why enterprises deploy voice AI on-prem
Three reasons come up again and again:
Data sovereignty and residency. Some contracts and jurisdictions require that specific data never leaves a physical or legal boundary you control. A cloud API streams audio to whoever runs it; on-prem keeps the boundary at your rack.
Compliance. Regulated buyers map deployments to frameworks like HIPAA, GDPR, and PCI. On-prem does not make you compliant by itself, but it makes the audit tractable: the data path starts and ends in infrastructure you can show an auditor.
Latency and control. When the models run on your network, you remove a network round trip to the vendor and inherit nobody’s rate limits. For a voice agent, response time is the product; people hand the conversational floor to each other in about 200 milliseconds (Stivers et al., 2009), and a second of silence reads as hesitation. This is also why the far end of this spectrum, running models on-device, exists: zero network, zero shared infrastructure.
The deployment spectrum
“Cloud vs on-prem” is really a spectrum of who operates what. Most enterprise rollouts land on one of these:
| Deployment | Where the models run | Who operates them | Data path |
|---|---|---|---|
| Cloud API | Vendor’s cloud | Vendor | Audio leaves your network |
| VPC (private cloud) | Your cloud account, your region | You (vendor-supported) | Audio stays in your cloud account |
| On-premise | Your data center | You | Audio stays in your network |
| Air-gapped on-prem | Your data center, no internet route | You | Nothing leaves, by construction |
| On-device | The endpoint itself | You | Nothing leaves the hardware |
Two notes that save buyers time. First, VPC is not on-prem: the hardware is yours, the data center is not. It is usually enough when the constraint is contractual. Second, ask every vendor what “on-prem” includes for them. Some mean a container you run that still phones home for licensing or updates. That is a connected on-prem, and it disqualifies itself for air-gap requirements.
How to evaluate an on-prem voice AI vendor
The questions that separate real on-prem offerings from marketing ones:
- Is the air gap real? Can inference run with no outbound connectivity at all, or does something (licensing, telemetry, a required cloud call) leave the network mid-conversation?
- Are the on-prem models the same as the cloud models? Some vendors ship you a different, older model than the one on their API. If the demos you loved were run on the cloud model, ask which binary you are actually buying.
- What latency do you get in your environment? Vendor numbers are measured on vendor hardware. Ask for a benchmark on yours, at your concurrency, at the 99th percentile, not the median.
- Who updates the models, and how? On-prem should not mean frozen. Ask how upgrades are delivered, whether you can pin versions, and what a model swap does to your tuned prompts and voices.
- What is the compliance paperwork? SOC 2 Type II, a signable BAA, a published DPA, data residency terms. If a vendor cannot produce these, the deployment model will not save the deal.
- Does the whole pipeline come indoors? A TTS model that runs on-prem next to an STT model that does not is a deployment with a hole in it. Ask about the pipeline end to end, including turn detection.
How Cartesia does on-prem
Cartesia’s models are built on state space models, an architecture designed for real-time inference, and the same models are available across cloud, on-premise, and on-device. Concretely:
- Sonic, our text to speech model (sub-90ms model latency, ranked first on third-party blind listening tests, 44 languages), deploys on-premise inside your data center, including air-gapped environments, in your own VPC on AWS, GCP, or Azure, or via OEM licensing to embed directly in your product.
- Ink, our streaming speech-to-text model with native turn detection, has the same deployment options.
- For ServiceNow’s AI Voice Agents, Cartesia provides the voice models, inference engine, and orchestration as a fully air-gapped on-prem deployment, alongside SOC 2 Type II, HIPAA, GDPR, and PCI compliance.
- Rasa bundles Cartesia’s on-premise deployment options into its enterprise offering, and Blue Machines co-developed a deployment architecture with us for regulated environments.
- At the far end of the spectrum, Edge is our open-source (Apache 2.0) library for running state space models on-device.
On-prem and OEM deployments are available under enterprise contracts; talk to us about your requirements, or start in the cloud on self-serve pricing and move inside later. Because the models are the same, moving is a deployment change, not a rebuild.