Is this TTS model good?

Most of our interactions with digital systems still happen through typing into boxes or tapping on screens. It works, but it is not exactly frictionless. Voice can make that interaction feel faster, more natural, and closer to the way people already communicate. If we can make it work well, we can unlock a more interactive way to learn, get help, move across languages, and get things done more seamlessly.

When people interact through voice, the generated speech becomes the experience. Small differences in delivery can change whether the interaction feels helpful, natural, or broken. That is what makes text-to-speech such an important piece of the stack: it turns written input into spoken output, and it shapes the feel of the entire interaction. Voice, though, is not a one-size-fits-all experience.

For example, when someone says an audio generated by TTS model is “good,” they might mean one or several things:

This multidimensionality is why TTS evaluation is tricky and as models improve, the evaluation shifts away from obvious defects like mispronounced words, garbled audio, or background noise towards more nuanced aspects. This blog post is a tour of the failure modes that keep showing up when we try to answer a deceptively simple question:

Is this TTS model good?

1. TTS: The one-to-many mapping problem

A modern TTS model does more than map text → audio. It must decide how to speak.

Text-to-speech is a one-to-many problem: there are infinite ways to read a transcript depending on the speaker, emotion, pace, emphasis, and context.

For example, “I can’t believe it” can be correct in four very different contexts:

• Excited: Someone just heard unexpectedly good news.
• Angry: Someone is reacting to a frustrating betrayal.
• Sad: Someone is trying to process a painful disappointment.
• Sarcastic: Someone is pretending to be impressed, but means the opposite.

Now permute the contexts. If the “excited” audio played in the angry scenario, or the sarcastic audio played in the sad one, every word would still be correct, but the speech would feel wrong.

Another example of different context but same text:

The first audio sounds excited, as though the speaker is looking forward to a big event. The second sounds more nervous or scared, as though the speaker is anticipating something stressful, like a difficult exam.

This variability in speech makes text-to-speech more difficult than simply converting words into sound. A good system must decide not only how each word should be pronounced, but also where to pause, which words to emphasize, how quickly to speak, and what emotion the voice should express. In many cases, the text alone does not provide enough information to make those decisions, so the system may need additional context or instructions about the desired speaking style.

Before we get to those subtler choices, though, evaluation usually starts with the most basic question: did the model say the intended words?

2. Saying the right words is only the first layer

WER (Word Error Rate): This is one of the most common metrics used for TTS models. You synthesize audio for an input text using TTS model, transcribe it with ASR (Automatic Speech Recognition) model, and compare the transcript to that input text.

Let’s look at some examples.

Both of these audios say all the words correctly (aka: Same WER(word error rate)) but most of the folks would naturally prefer the second audio.

So, WER can be directionally helpful. But once intelligibility is largely solved, the deltas shrink into noise.

• WER saturates: As demonstrated from the example above, the two models can differ meaningfully in perceived quality while landing within negligible differences in WER (especially on clean, read-style test sets).
• Tied to ASR quality: WER is only as reliable as the speech recognizer used to compute it. If the ASR model is weaker on certain accents, languages, noisy clips, expressive speech, or domain-specific words, the TTS system can be penalized for ASR mistakes rather than actual synthesis errors.
  • Conversely, a strong ASR can sometimes “correct” or infer unclear speech, making a flawed sample look better than it sounds to humans.

WER is a necessary metric for many use cases, especially those involving alphanumerics that are not words. But WER is rarely sufficient for enterprise-grade voice agents.

3. Contextual correctness is another challenge

Proper nouns are worse: names of people, places, products, companies, drugs, and fictional terms often lack canonical pronunciations and even humans disagree (especially cross-lingually).

A good eval needs to detect when the model picked the wrong interpretation even if it sounds fluent.

4. Text normalization is its own evaluation problem

Pronunciation is only one place where context matters. Before the model can pronounce anything, it often has to decide how written text should be expanded into speech.

Many inputs are ambiguous:

• “$1.05” (one oh five vs one dollar and five cents)
• “3/4” (three quarters vs three fourths)
• “IV” (four vs I-V)
• “St.” (street vs saint)
• “No. 5” (number five vs “no. five”)

A fun example is date in different locales. For the following text, US English expects July 8th while British English expects the 7th of August!

Multiple normalizations can be correct depending on context, and reference transcripts frequently encode a different choice than your product wants. If you don’t separate normalization preference from model error, your evaluation scores become incoherent.

5. Automated quality scores are helpful, but incomplete

Transcript accuracy is only the floor. Once a model says the right words, the remaining quality gap is usually in prosody, expressiveness, style, and stability: the parts that make speech feel natural, appropriate, and usable.

Automated quality predictors try to measure this at scale. Tools like DNSMOS, NISQA, and UTMOS listen to generated clips and assign quality scores, which makes them useful for catching regressions across thousands of samples.

But these scores are still proxies. They do not really ask, “Did this clip work for this situation?” They ask, “Does this clip resemble the kinds of audio I learned to score?”

That distinction matters because the failures are often non-obvious:

• Domain mismatch: A quality predictor may work well on the kind of speech it was trained on, but become unreliable when the audio comes from a different setting.
  • Ex: UTMOSv2 was specifically developed to predict naturalness MOS for high-quality synthetic speech (predominantly English). It was not designed as a universal metric for codecs, noisy speech, multilingual, singing, or conversational audio.
• Calibration drift: A score that used to separate older models may stop being meaningful once newer models fail in subtler ways. This often means that your automated quality predictors also need retraining to stay relevant.
• Blind spots: The predictor may notice noise or distortion but miss awkward emotion, unnatural pacing, streaming glitches, or gradual voice drift.
  • For example, in this study of automated MOS predictors, the same speech was generated with increasing amounts of wrong Japanese pitch accent. Humans drop the samples by 1.84 MOS points. Every automated model moves by less than 0.1. Some predictors actually score the heavily corrupted speech slightly higher.

In other words, these scores are useful for catching obvious regressions at scale, but they are not a replacement for actually understanding what makes a clip good.

At this point you often “have to” move to human evals but that opens a different set of traps.

6. Human evals: the gold standard?

Human listening tests are still the gold standard, but the scores are not a stable measurement. Judgments vary with listener background, native-language status, the wording of instructions, anchors and baselines included in the set, playback environment (headphones vs speakers) / loudness normalization, session length (fatigue) and more.

Two common human eval formats are MOS (Mean Opinion Score) ratings and preference ratings. In a MOS test, listeners rate each clip independently, often on a 1–5 scale for quality or naturalness. In a preference test, listeners compare two or more clips and choose which one they prefer. MOS gives an absolute-looking score, while preference tests are more directly comparative, but both are shaped by the instructions, listener pool, and presentation setup. Regardless of which you are using, human evals suffer from the following:

6.1 Listener responses are context-dependent

Listeners do not hear clips in a vacuum. Even if everyone hears the same sentence, they may imagine different situations around it. A line like “That’s great” could sound sincere in one context, sarcastic in another, and flat or confusing in a third.

That means the same audio can receive different scores depending on what the listener thinks the voice is supposed to be doing. If the task does not clearly specify the intended scenario, people end up judging against their own private expectations rather than a shared target.

6.2 Comparisons are sensitive to presentation details

Small seemingly “irrelevant” choices can dominate results:

• Loudness / peak normalization strategy: A louder sample can feel clearer, more confident, or higher quality even if the model itself is not better.
• Sample rate, codec, and resampling: Compression or resampling artifacts can make one system sound worse for reasons unrelated to the model synthesis quality.
• Prompt text, UI context, surrounding samples: The text and presentation frame what listeners expect, which can shift what they notice or prefer.
• Trimming silence / breaths / leading latency: Small edits around pauses, breaths, or startup delay can change whether a sample feels natural, awkward, or responsive.

If these aren’t locked down, you might be evaluating the audio pipeline rather than the model.

6.3 Drift is real (even if the model is stable)

Over time, listeners change too:

• Adaptation: after many synthetic clips, listeners recalibrate what they notice (more sensitive to artifacts, or more forgiving)
• Fatigue: Audio annotations are exhausting and judgment quality drops as sessions get longer
• Environment variance: what pops on headphones can disappear on phone speakers, and vice versa.

Treating human MOS as immutable ground truth will punish you later.

6.4 Preferences and tastes vary

A recurring pain: people don’t want the same thing. Even among attentive listeners, preferences differ for:

• Voice identity / timbre: Some listeners prefer bright, crisp voices; others prefer warmer, deeper, or more textured voices.
• Speaking rate: A faster pace can feel efficient to one listener and rushed or stressful to another.
• Accent and dialect: A voice can sound familiar and natural to one audience but distracting or mismatched to another.
• Emotional intensity: Some listeners like expressive delivery, while others find the same performance overacted or tiring.
• Age / persona cues: Listeners may infer different personalities from the same voice and prefer different personas for different products.

Sources: Southern accent and New York accent

This creates a real possibility that two systems can each be “better,” depending on which listener you ask. The eval question becomes: “better for whom?”

7. Long-form and streaming expose different failure modes

Most TTS evals use short clips because they are easier to generate, listen to, and score. But real products often ask a voice to keep going: reading an article, narrating a chapter, handling a support call, or carrying on a back-and-forth conversation. That is where new problems appear.

In long-form audio, a clip does not only need to sound good sentence by sentence. It also needs to stay coherent over time. Common drift patterns include:

• Speaker identity changes
• Accent drift
• Speaking rate changes
• Emotional consistency across paragraphs

Streaming adds another layer. Streaming means the system starts playing audio before the full response has been generated. Instead of waiting for the whole paragraph or answer to be ready, the model produces speech in small pieces and sends them to the listener right away. This is what makes a voice assistant feel responsive: you hear it start talking quickly, even while the rest of the sentence is still being created.

That speed is useful, but it comes with a trade-off. Because the model has to start speaking before it knows exactly what is coming next, the streamed version can fail in ways that an offline clip does not. For example:

• Awkward joins between chunks: each piece sounds fine alone, but the transition feels stitched together.
• Repeated syllables or words: the voice may repeat itself around a join: “look-look-looking” or “is is scheduled.”
• Cut-off sounds: a word may start or end too abruptly, like “stop” becoming “top.”
• Choosing the wrong delivery too early: the model may start with the wrong pace, emotion, or emphasis and struggle to recover.

The broader point is that short clips can hide failures that only show up when TTS is generated live. Long-form tests ask whether the voice stays consistent over time. Streaming tests ask whether the voice still sounds natural when it has to start quickly, before the full response is known.

These failures are not random; they usually appear when the evaluation setup stops matching the way the product is actually used. That is why the next layer is domain: the same voice that sounds strong on generic read speech can fall apart when the text carries specialized conventions, formulas, names, or pacing expectations.

8. Domain-specific text breaks assumptions

Every domain (e.g. legal, medical, gaming, math, poetry, religious text, customer support) has different expectations:

• formatting and normalization
• rare word handling
• cadence expectations
• correctness constraints vs style constraints

A model can perform at “SOTA” quality in one domain and be audibly broken in another.

Notice how the cadence of pronouncing H₂O is broken and the TTS does not pronounce the “→”. A chemistry major would read it out as “yields” or “produces”.

Domain is one axis of variation. Language and locale add another, and they often break the eval stack in entirely different ways.

9. Multilingual introduces unique challenges

Multilingual TTS is not just the English problem run N times. Each language breaks a different part of the eval stack. And you can’t score a language you don’t speak, so native listeners stop being optional. Here’s what specifically breaks:

Your automated metrics quietly stop working. WER inherits the biases of the ASR model used to compute it: recognizers are often weaker on non-English accents, scripts, and low-resource languages, so the TTS system can get penalized for the recognizer’s mistakes rather than its own. Learned quality predictors have the same problem. Tools like UTMOS were trained largely on English, so off-distribution they may measure very little. The automated layer you rely on to scale English evals becomes least reliable exactly where you most need extra throughput.

Japanese is harder: kanji often doesn’t determine pronunciation, especially for names. For example, 上手 has several valid readings (jōzu, uwate, or kamite), and without context the model may pick a plausible but wrong one.

The same kind of trap recurs everywhere in different ways: tone in Mandarin (wrong tone, wrong word), unwritten short vowels in Arabic, schwa-deletion in Hindi. Scoring any of these correctly requires native speaker knowledge.

Correct language, wrong accent. A model can produce fluent, fully intelligible speech in the wrong locale, e.g. Canadian French words that come out with Parisian intonations or a European Spanish voice that drifts toward Mexican Spanish pronunciations mid-clip.

Such examples would pass every intelligibility and expressiveness check but still be wrong. There’s a mirror-image failure on cross-lingual voices: a Hindi speaker’s pronunciation of English words should keep its Indian accent, and “fixing” it to General American is an error, not an improvement. This is an entire correctness axis a monolingual eval never sees.

One number hides the gap. Because failures concentrate in different dimensions per language, e.g. prosody in one, pronunciation in another, locale-specific normalization in a third, a single aggregated multilingual score averages away the very signal you would act on. “Better multilingually” is only meaningful per-locale and per-dimension.

Overall, multilingual eval pushes you away from a single global score and toward a native-speaker-in-the-loop view of quality: broken out by locale, by failure mode, and by the product expectations that actually matter.

Closing: TTS is more than 1 dimension

TTS evaluation is hard because “good” is not one thing. It depends on multiple aspects. An audiobook voice, a navigation voice, and a customer support agent should not be optimized for exactly the same thing.

At Cartesia, we care especially about conversational speech: voice agents, interactive assistants, and other real-time settings where the system has to respond quickly and still feel natural. In those products, a beautiful offline clip is not enough. The voice has to work inside a live conversation at enterprise scale, so that our customers get the results they need from voice agents.

That’s why we optimize for:

So, the point is to define the product experience you want, then build multi-dimensional evals that measure whether the model actually delivers it. For us, that means optimizing for speech that makes real-time conversations feel fast, clear, and natural.