P99 latency is the response time that 99% of requests beat. If an API’s P99 is 400 milliseconds, 99 requests in a hundred come back in 400 ms or less and one takes longer. P95 is the same idea at 95%: one request in twenty is slower. Both describe the tail of the distribution, the slow requests that an average smooths away.
The tail matters more than its size suggests. A user who makes one request rarely notices it. A caller who talks to a voice agent for twenty turns probably will. This guide covers how percentiles work, how to compute them, the mistakes that make them lie, and what they mean for real-time voice AI.
P50, P95 and P99 at a glance
| Percentile | What it tells you | Requests slower than it |
|---|---|---|
| P50 | The median, a typical request | 1 in 2 |
| P90 | The start of the slow tail | 1 in 10 |
| P95 | The usual target for dashboards and SLOs | 1 in 20 |
| P99 | Rare slowdowns: pauses, retries, queueing | 1 in 100 |
| P99.9 | Outliers that only show up at high volume | 1 in 1,000 |
Higher percentiles need more samples to mean anything. A P99 from 100 requests rests on a single request. Aim for at least a few hundred samples for P95 and a few thousand for P99, and print the sample count next to the number.
Why the average hides the tail
Take 100 requests. 94 finish in 200 ms and 6 take two seconds. The mean is 308 ms, which no request actually took. The median is 200 ms, which looks great. P95 and P99 are both 2,000 ms, which is what six of your users saw.
Real latency distributions are lopsided like this. Most requests take the fast path, and a minority wait behind a slow disk, a cold cache, a garbage collection pause or a full queue. An average blends the two groups into a number that describes neither. A median ignores the slow group entirely. Percentiles keep it visible.
How to calculate P95 and P99
Sort the measurements from fastest to slowest and read the value 95% or 99% of the way along. With 200 samples, P95 is the 190th value and P99 is the 198th. That’s the nearest-rank method:
import math
def percentile(samples, p):
"""Nearest rank: the smallest sample with at least p% of samples at or below it."""
ordered = sorted(samples)
rank = math.ceil(p / 100 * len(ordered))
return ordered[max(rank, 1) - 1]
latencies_ms = [200] * 94 + [2000] * 6
print(percentile(latencies_ms, 50)) # 200
print(percentile(latencies_ms, 95)) # 2000
print(percentile(latencies_ms, 99)) # 2000NumPy interpolates by default, so numpy.percentile can return a value between two samples. Pass method="inverted_cdf" to get nearest rank. On small samples the two methods can differ noticeably, which is one more reason to report the sample count.
Monitoring systems usually don’t keep every sample. They count requests into latency buckets and estimate the percentile from the histogram, as Prometheus’ histogram_quantile() does. The estimate is only as precise as the buckets around it, so put bucket boundaries near the latencies you care about.
Three mistakes that make percentiles lie
Averaging percentiles. Say one server handles 900 requests, all in 200 ms, and a second handles 100, ten of which take a full second. The P95s are 200 ms and 1,000 ms, and their average is 600 ms. The real P95 across all 1,000 requests is 200 ms. Prometheus’ documentation says the same: quantiles computed separately can’t be aggregated. Merge the histograms or raw samples, then compute.
Adding stage percentiles. A voice agent runs speech recognition, a language model and text to speech in sequence. Suppose recognition always takes 300 ms, the LLM takes 500 ms but 1,500 ms on one call in ten, and TTS takes 150 ms but 600 ms on one call in ten. Adding the stage P95s gives 2,400 ms. The actual P95 of the whole turn is 1,950 ms, because the two slow cases rarely land on the same turn. Measure the end-to-end time directly instead of estimating it from parts.
Measuring with a client that waits. A load generator that sends the next request only after the last one returns sends fewer requests exactly when the server is slow, so the slow periods are undercounted. Gil Tene calls this coordinated omission, and his load generator wrk2 exists to avoid it by sending at a fixed rate. For production numbers, measure real traffic rather than a test client.
Why the tail decides how a voice agent feels
Conversation runs on tight timing. A review of turn-taking research by Levinson and Torreira finds that gaps between turns are on the order of 200 ms. In a study of telephone calls, Kendrick and Torreira found that responses after gaps longer than about 700 ms were mostly refusals and other dispreferred answers, and that even gaps over about 300 ms made a plain acceptance less likely. Listeners read a pause as meaning something, so to a caller an agent that pauses for 900 ms can sound reluctant, not just slow.
Now multiply by turns. If each turn independently has a 5% chance of being slower than P95, a 20-turn call has a 64% chance of hitting at least one of them (1 − 0.95²⁰). Even the P99 shows up on about 18% of 20-turn calls. The median describes a typical turn; the tail describes a typical call. Google’s Jeff Dean and Luiz André Barroso made the same point about large services in The Tail at Scale: when one user action fans out into many requests, rare slow responses become common.
What to measure in a voice pipeline
Measure what the caller hears. For a voice agent, that’s the time from the end of the caller’s speech to the first audible sample of the reply, recorded on the caller’s side of the line. Then break it into stages so you know where the tail comes from:
| Stage | What to time | Common tail causes |
|---|---|---|
| Turn detection | End of speech to the turn being declared over | Hesitations, background noise, long pauses |
| Speech to text | Turn end to final transcript | Long turns, network jitter |
| LLM and tools | Transcript to first token, including tool calls | Slow tools, long prompts, provider queueing |
| Text to speech | First text sent to first audible audio | New connections, long first sentences |
| Network and phone | Audio leaving your server to reaching the caller | Carrier routing, jitter buffers |
Time to first audio (TTFA) is the TTS number to watch, because a streaming model starts playback before the whole sentence is ready. Leading silence counts too: a chunk that starts with 300 ms of silence is 300 ms more waiting for the caller. The script below times the first bytes, so check your audio for leading silence separately.
Here’s a script that measures TTFA percentiles for Cartesia’s Sonic text to speech. It schedules 205 requests half a second apart, discards the first five, and times each from its scheduled start to the first audio bytes. Timing from the schedule rather than from when the request went out means a slow response that delays the next request shows up in the numbers, which avoids the coordinated omission described above:
import math
import os
import time
from cartesia import Cartesia
def percentile(samples, p):
"""Nearest rank: the smallest sample with at least p% of samples at or below it."""
ordered = sorted(samples)
rank = math.ceil(p / 100 * len(ordered))
return ordered[max(rank, 1) - 1]
# max_retries=0: a retried request would hide a failure inside a slow sample.
client = Cartesia(api_key=os.environ["CARTESIA_API_KEY"], max_retries=0)
interval = 0.5 # seconds between scheduled requests
ttfa_ms = []
failures = 0
first = time.perf_counter() + interval
for i in range(205):
# Each request has a fixed start time. If a slow response makes us late,
# the wait counts against latency instead of quietly thinning the sample.
scheduled = first + i * interval
time.sleep(max(0.0, scheduled - time.perf_counter()))
got_audio = False
try:
with client.tts.with_streaming_response.generate(
model_id="sonic-3.6",
transcript="Your order shipped this morning and should arrive on Thursday.",
voice="6ccbfb76-1fc6-48f7-b71d-91ac6298247b",
output_format={"container": "raw", "encoding": "pcm_s16le", "sample_rate": 16000},
language="en",
) as response:
for chunk in response.iter_bytes():
if chunk:
got_audio = True
if i >= 5: # the first requests pay for connection setup
ttfa_ms.append((time.perf_counter() - scheduled) * 1000)
break
except Exception:
pass
if not got_audio:
failures += 1
print(f"{len(ttfa_ms)} timed requests, {failures} failed or returned no audio")
if ttfa_ms:
for p in (50, 90, 95, 99):
print(f"p{p}: {percentile(ttfa_ms, p):.0f} ms")
print(f"max: {max(ttfa_ms):.0f} ms")Install the SDK with pip install cartesia and set CARTESIA_API_KEY from the dashboard. Run it from the region your servers run in, at the time of day your traffic peaks, because a laptop on home Wi-Fi measures the Wi-Fi. The SDK retries failed requests twice by default, which would fold a failure and its retry into one slow-looking sample, so the script turns retries off and counts failures, including responses with no audio, separately. Don’t leave them out of the report: a request that never answered is the worst latency of all. For a live agent, time the WebSocket path you actually ship, since it keeps one connection open across turns.
What public benchmarks show
Coval runs independent text to speech, speech to text and speech-to-speech benchmarks against public APIs every 30 minutes and publishes medians, percentiles and sample counts. Its methodology counts leading silence in TTFA and excludes connection handshakes.
On its 30-day text to speech board, read on October 15, 2026, the gap between median and P95 varied widely across 33 models with at least 260 samples each. For the typical model, P95 was 1.38 times the median. At the extremes, one model’s P95 was 1.1 times its median, and another’s was more than seven times its median: 624 ms typical, 4,751 ms for one request in twenty.
The lesson is the one from the start of this guide: two models with similar medians can feel very different on a call. Compare P95s, with sample counts, measured the way your users experience them.
Lowering tail latency
- Keep connections open. A new HTTPS request pays for TCP and TLS setup again, and Cartesia’s docs note that this overhead is often on the same order as the time to the first audio byte itself. A WebSocket that stays open for the call pays it once.
- Cache what repeats. Greetings, hold messages and sign-offs don’t need a fresh generation on every call. Cache them as raw audio and splice them into the live stream.
- Start speaking sooner. Stream LLM tokens into text to speech as they arrive instead of waiting for a whole paragraph.
- Put a deadline on tools. A tool call that takes two seconds adds two seconds to the turn, every time. Set a timeout, and have the agent say what it’s doing while it waits.
- Hedge the slow requests. The Tail at Scale describes sending a backup request once the first has been outstanding longer than the usual P95, and using whichever answers first. Waiting until P95 limits the extra load to about 5%. Only hedge requests that are safe to send twice, like generating the same audio again, never a tool call that writes.
- Run the models close to the call. If network distance or shared infrastructure is your tail, on-premise deployment puts the models next to your telephony.
If you’d rather not assemble and tune the pipeline yourself, Cartesia Managed Agents run turn detection, Ink speech to text, the LLM and Sonic on a phone number, and you can build one in the playground and call it. If your own pipeline is slow, the fix is usually in whichever stage your measurements point to, and the script above is the place to start.