Why Precision / Recall Matters So Much

Two ways to be wrong

Precision vs recall is obvious if you evaluate ML models for a living. For everyone else, not so much.

It’s best understood with an example.

Let’s say you’re building a RAG pipeline. And you’re really trying for highly reliable retrieval from your vector store. One of the first questions: how many chunks should you fetch (your top_k)?

If you set it to fetch 3 chunks they’re all likely to be highly relevant. But you might miss relevant chunks ranked 4th or 10th. That means relevant material got left out.

You could try and fix this by increasing your top_k to 50. With that setting you catch everything relevant, but there was also plenty of irrelevant stuff which increases the odds that the LLM returns “noisy” info that has some non-relevant information, even though you retrieved all the relevant chunks from your vector store.

This is a trade off, and it’s referred to as Precision vs Recall.

The top-3 retrieval has high precision because what you got back was high relevance. It was on point. But it also has low recall because you missed stuff.

On the other hand, the top-50 retrieval has high recall (you caught everything) but lower precision (you caught a lot of junk too).

Precision and recall show up everywhere a system has a detection job. A system that detects something has a binary job — detect X (true) or not (false). In doing so it can be right or wrong.

We will dive more into this when we talk about the Confusion Matrix later.

And yes this is confusing. So it’s better to understand why we bother with Precision vs Recall at all. Short answer: measuring “accuracy” alone misleads.

Why accuracy alone misleads

Suppose a detector sees 100 examples. Ten contain the event it should catch. But the detector returns “no event” every time — for all 100. The system scores 90% on this test because it is 90% accurate for “no event.”

But it is 0% accurate for the events that were in the examples and ought to have been caught.

A security guard at a building entrance: if only 1 in 100 visitors is a threat, a guard who waves everyone through is “99% correct” for letting the right people in. But if the guard is measured on success at keeping the wrong people out then the guard is 0% correct because their entire job is to catch the 1 bad visitor.

So, when you reflect on this, you can see that when a dataset is skewed heavily in favour of one outcome, then a model that just randomly guesses across all its turns will return a high score just by chance — and will be accurate on paper. But it fails at actually predicting things it’s meant to predict.

Identify the Event

Since this is confusing, I find it helps to not analyse Precision vs Recall until a system has a defined detection job — precision and recall are specific to the job to be done.

You start with your True Positive case.

For a spam filter, the event is “this email is spam” (True Positive). For a smoke detector, it is “there is smoke.”

This is “the Event” to flag. So far, so good.

But what if the system has two tasks? For example, for an AI Speech to Text model that also handles automatic turn detection, “the Event to flag” may be “the user started speaking” as well as “the user stopped speaking.” Note that stopped speaking is not the same as “finished their turn.” Each of these events will need their own precision/recall scores.

A positive means the system says the event happened — a label for one side of the detection task, not a value judgment.

This grid is called a confusion matrix. It maps every possible detection outcome for a given event: what actually happened vs what the system said.

Ask four questions before choosing a metric:

• What event should the system detect?
• What counts as the system firing?
• What does a false positive look like?
• What does a false negative look like?

Start with the denominator

I found starting with the denominator useful in understanding the difference between precision and recall.

Precision

Precision denominator = everything the model predicted as positive (correct + incorrect predictions). True positives + false positives.

For Precision’s numerator, count how many of the predictions were correct. The true positives.

Which gives us the following formula:

precision = true positives / (true positives + false positives)

Consider a “speech started” event. There are 100 audio samples. The filter flags 80 audio samples as speech started, but 60 of those actually had speech started events.

So True positives are 60. False Positives are 80 - 60 = 20. We then get:

precision = 60 / (60+20) = 75%

Low precision incorrectly identifies other sounds, or the sound of silence, as speech.

So precision is a ratio of what the model got right to what it got right and what it got wrong.

Recall

Recall denominator is the total count of true positives that actually exist (caught + missed). True positives + false negatives. So the denominator includes all true events whether the model found it or not.

The numerator is the same as for precision — we count true positives.

recall = true positives / (true positives + false negatives)

So recall is a ratio of what the model got right to what it got right and ought to have caught (which is the same as what it wrongly excluded).

Using the same speech detection example, True positives are 60. False Negatives are 0 because the model caught all of them and then some.

We then get:

recall = 60 / (60+0) = 100%

This is not illustrative of recall’s value, so let’s change the example. Let’s say there are 100 audio samples. The model flags 50 audio samples as speech started, but 60 of those actually had speech started events.

Then true positives are 50 and false negatives are 60 - 50 = 10. Those are 10 speech start events the model missed.

Then we get:

recall = 50 / (50+10) = 83%
precision = 50 / (50 + 0) = 100%

Note that in this updated example precision is 100% because false positives are zero.

Note also how these are different from a simple “accuracy” measure:

accuracy = (true positives + true negatives) / all samples

Use accuracy when each decision has similar cost and the evaluation set represents production. But when the target event is infrequent, or the two mistakes (the two false outcomes) have different costs, reporting against the confusion matrix makes all the difference for performance.

And if you want to nerd out on the mathematical formulae and their derivatives, here is the full Wikipedia entry!

The precision/recall trade-off

From the worked examples above, you can see a real trade-off. Push one up, the other tends to drop.

The retrieval example at the top shows why: retrieve more documents and recall goes up, but precision drops. Retrieve fewer and precision goes up, but recall drops.

In practice, most models produce a confidence score on their detections — a number between 0 and 1 that represents how sure the model is. A threshold is the cutoff you choose: “flag anything above 0.7 as positive.”

It works like a volume dial on a hearing aid. Turn the sensitivity up and you hear every conversation — but also the air conditioner, the traffic, the neighbor’s TV. Turn it down and you hear only loud, clear speech — but you miss quiet voices.

Raise the threshold and the model flags fewer false positives. Precision rises.

Lower the threshold and the model flags more cases correctly which means it lowers its false negatives. Recall rises.

Models that live along the Pareto frontier, like Cartesia’s Ink-2, balance these tradeoffs optimally.

Apply this to Voice AI

Voice AI contains several detection and prediction jobs. Do not grade all of them with one accuracy number.

Voice activity detection (VAD) asks whether a window of audio contains speech. A test set can contain far more silence frames than speech frames. Predicting silence every time looks accurate while missing all speech. Measure speech precision and recall. Report how often the detector correctly flags speech and how often it misses it.

Turn detection decides when a speaker takes or yields the conversational turn. Its start and end events need separate metrics. Below we use two Ink-2 ASR turn events.

turn_start: did the user begin speaking?

The target event is “the user began speaking.” The system fires when it emits a turn_start event.

• A false positive: it emits turn_start for keyboard noise, background speech, microphone bleed, or the agent’s own audio.
• A false negative: the user begins speaking and the system misses it.

Low turn_start precision makes the system react to non-user audio. If turn_start stops playback, false positives create false interruptions.

Low turn_start recall means real speech can fail to reach the rest of the pipeline.

turn_end: did the speaker finish?

The target event is “the user finished their turn.” The system fires when it emits a turn_end event and the application may respond.

• A false positive: it emits turn_end while the user plans to continue.
• A false negative: the user finished, but the system keeps waiting.

Low turn_end precision can cut the user off after a pause to think, correct a sentence, or take a breath. Low turn_end recall creates dead air after the user is done.

For both events, define what counts as a match between a predicted timestamp and a labeled timestamp. Define the target speaker, how to label overlapping speech and background audio, and what “finished” means before calculating the scores.

Choose the metric around the failure

• Report turn_start and turn_end precision and recall separately.
• Choose the threshold based on which error costs more in the product.
• Inspect false-positive and false-negative examples, not only aggregate scores.
• Use accuracy as context, not the whole evaluation.

The next time a model reports 90% accuracy, ask what it detected, how often that event occurred, and which mistakes the score hides.