Word error rate (WER) counts differences between a transcript and a reference. It can help detect missing or incorrect words. It cannot tell you whether a voice is good for your product.
Two generated clips can say exactly the same words and still create very different experiences. One sounds reassuring; the other sounds delighted that your flight was canceled. The transcript passes. The conversation does not.
Our approach, explained in Is this TTS model good?, starts with the experience we want people to have. For real-time voice agents, that means understanding the response the first time, hearing delivery that fits the situation, and having a conversation without awkward delays or interruptions. WER is one diagnostic within that evaluation, not a score to optimize in isolation.
What word error rate actually measures
WER uses the minimum number of word edits needed to align a transcript with its reference:
WER = (substitutions + deletions + insertions) / reference wordsMultiply by 100 to express it as a percentage. In this example, the transcript substitutes “Thursday” for “Tuesday”:
Reference: please move my appointment to next Tuesday morning
Transcript: please move my appointment to next Thursday morning
WER = 1 / 8 = 12.5%That is one word error, and a potentially wrong appointment. WER gives it the same weight as any other substitution. The consequence belongs in the product evaluation.
Microsoft’s WER documentation explains the calculation; tools such as JiWER compute the alignment. Use consistent text normalization and reference rules. For corpus WER, divide total edits by total reference words rather than silently averaging each clip’s percentage. Insertions can produce WER above 100%, and an empty reference needs a documented scoring convention because the usual formula divides by zero.
The calculation is the same in two different evaluation setups:
| Evaluation | What you compare | What the result can tell you |
|---|---|---|
| Speech recognition (STT/ASR) | The recognizer’s transcript against a human reference | How often recognized words disagree with the reference |
| Text-to-speech (TTS) | An ASR transcript of generated audio against the intended words | Whether synthesis and recognition together preserve those words |
In the second setup, you are evaluating speech through another model. An apparent TTS error may come from the recognizer. A clean transcript may also conceal unclear speech. You need to listen to establish which happened.
Why WER can mislead TTS evaluations
Identical words can sound very different
Text does not specify a single correct performance. “I can’t believe it” can express excitement, anger, sadness, or sarcasm. Recognizing every word does not tell you whether the delivery fits the situation.
The paired audio examples in our evaluation post make this audible: clips can have the same WER while differing in delivery. Once models reliably say the intended words on a test set, small WER differences may tell you little about the quality differences listeners care about. More decimal places will not recover information the metric never measured.
The recognizer can both invent and hide errors
An ASR model may struggle with an accent, a language, or an unfamiliar name even when the generated speech is clear to its intended listeners. That raises WER without establishing a synthesis failure.
The opposite can happen too. A recognizer may infer an unclear word from context and return the intended transcript. A human trying to catch an account code over the phone may not be so lucky.
Keep the recognizer and scoring policy fixed for regression tests, and listen to flagged clips before attributing errors to TTS. Sample passing clips too: reviewing only failures will not reveal the cases where ASR repaired the output. A fixed recognizer makes comparisons more consistent; it does not remove its blind spots.
Correctness depends on context
A transcript can contain “read” whether the model pronounced it in the present or past tense. The words match, but the pronunciation can be wrong for the sentence.
Written dates expose a related problem. “07/08/2026” can mean July 8 in a US context or 7 August in a British context. The product must establish the intended locale before a reference can establish correctness. Neither interpretation should lose merely because the reference writer assumed the other one.
Some differences are harmless: “$25” and “twenty-five dollars” can represent the same amount. Others change the meaning. Define acceptable spoken forms for the task, then distinguish a formatting mismatch from the wrong date, amount, or identifier. Text normalization should not erase a mistake the caller would care about.
Where semantic WER helps
Semantic WER aims to distinguish wording differences that preserve meaning from errors that change it. “Twenty-five dollars” and “$25” can mean the same thing; “fifteen dollars” does not. That makes a meaning-aware score useful alongside word-level WER, especially when literal transcript matching overstates harmless differences.
The scoring method matters: define what counts as equivalent and check that the evaluator catches changed names, amounts, and negations. Do not assume two implementations of “semantic WER” are comparable. Even a perfect meaning match says nothing about whether the voice sounds reassuring, rushes an address, or glitches between audio chunks. It adds a diagnostic, not a verdict on conversational quality.
Start with the conversation you want to support
Before comparing models, describe the situation. Who is listening? Which language and locale do they expect? Are they hearing an address over a phone connection or a short answer through headphones? What must they understand or do next?
For a support agent reading an appointment confirmation, you might require the caller to understand the date and time on the first hearing, a calm delivery, and enough pause between details to follow them. A lower aggregate WER does not establish any of those requirements by itself.
Build the evaluation around those requirements. For conversational TTS, we use dimensions like these:
| Dimension | Question to test | Evidence to collect |
|---|---|---|
| Correctness and clarity | Can listeners recover names, numbers, addresses, and codes correctly? | Listener transcriptions and checks of the critical details, with WER as a diagnostic |
| Contextual correctness | Is the pronunciation or spoken expansion right for this situation? | Review against explicit locale, domain, and acceptable-reading rules |
| Appropriate delivery | Does the pace, emphasis, and emotion fit the conversation? | Listening comparisons with the scenario supplied |
| Responsiveness | Does the caller wait too long, or get interrupted? | End-to-end timing and turn-taking observations in the agent |
| Consistency | Does the speaker remain stable across turns? | Review of complete conversations for changes in voice, accent, or delivery |
| Streaming behavior | Does live playback introduce joins, repeats, or clipped sounds? | Audio captured through the intended streaming and playback path |
Some of these tests evaluate TTS; others evaluate the whole agent. Keep that distinction when diagnosing a failure. An interruption may come from turn detection, and a long pause may come from a backend lookup rather than speech generation. The caller experiences all of them, but the fix depends on where the problem starts.
Listen in context, and test the live path
Human listening is indispensable, but a preference score is not universal ground truth either. Tell listeners what the voice is supposed to be doing. Recruit people who understand the language and locale, and report where their judgments differ rather than reducing every audience to one average.
For model comparisons, control playback level, codec, sample rate, and presentation order. Keep sessions short enough to limit fatigue. If you are testing the complete product instead, preserve its real phone or playback conditions and state that scope. Otherwise you can mistake a codec problem for a model problem, or remove a production problem while preparing the test.
Use both short clips and complete conversations. A polished offline recording cannot reveal every failure of speech generated while the response text is still arriving. Listen for abrupt chunk joins, repeated syllables, and delivery that starts in the wrong direction and cannot recover. Across longer conversations, check whether the voice or accent drifts.
Break results out by locale and failure mode. A model can improve on one language’s pronunciation while getting worse at another’s pacing. A global score can hide both changes. Automated quality predictors can help screen for regressions, but they also need validation on your languages and audio conditions; replacing WER with another single number does not solve the evaluation problem.
Use WER to investigate, not to declare a winner
WER remains useful for tracking word-level regressions and finding clips worth reviewing. For names and alphanumeric codes, accurate delivery can be essential. Keep those checks. Pair them with evidence that the speech is understandable and appropriate in the conversation where it will be used.
When a score improves, ask what changed for the listener. Did an address become easier to understand? Did a missing word return? Or did a formatting change make the reference and transcript agree? Those are different outcomes, even if the dashboard uses the same green arrow.
To hear the distinctions, start with the audio examples in Is this TTS model good?. Then test Sonic with your own conversational text and streaming setup. Choose the model that meets your product’s requirements across the dimensions that matter to your users. A transcript score alone cannot make that choice.