Definition
AI hallucination is when an LLM generates text that is nonsensical or unfaithful to the source it was given, or states things with no support. In a voice agent, it is a confident spoken answer the caller cannot check.
How AI hallucination works
An LLM produces text by predicting what plausibly comes next. It is trained to write fluent, likely text, not to check facts, so when it lacks the information it can still produce an answer that sounds right.
The survey by Ji and colleagues describes hallucinated text as nonsensical or unfaithful to the provided source content. It splits the problem in two.
- Intrinsic. The output contradicts the source. A document says returns are accepted for 45 days and the model says 30.
- Extrinsic. The output cannot be verified from the source. The model adds a policy the document never mentions. The survey notes that such text is not always wrong, but that it is usually treated with caution.
Causes include gaps in training data, ambiguous questions, retrieval that returned the wrong passage, and instructions that push the model to always give an answer. A model is not lying. It has no built-in sense of what it does not know.
Reducing it takes several measures together. Give the model the facts to use, through a knowledge base or a tool call, and instruct it to say when it does not know. Keep sensitive facts in systems of record and not in the model’s memory. Test for it with questions that have no answer in your material.
Why it matters for voice agents
A caller cannot see the source, cannot scroll back and cannot easily hold the agent to what it said. A wrong spoken answer about a price, a deadline or a medication is heard once and may be acted on. Voice also invites confidence: smooth speech makes a made-up answer sound as sure as a correct one.
Speech recognition adds a second route to wrong answers. If the recognizer mishears “fifteen” as “fifty”, the model can answer the wrong question perfectly. Confirming high-stakes details by reading them back catches that.
Practical controls for voice agents:
- Tell the agent to say “I don’t have that” and offer a transfer, in place of guessing.
- Fetch live facts, such as order status, with a tool, and do not rely on memory.
- Add guardrails for topics where a wrong answer has real cost, such as legal or medical advice.
- Review recorded calls for answers that are not in your documents.
Example
An illustrative failure. A caller asks if a service plan covers water damage. The documents do not mention it. An agent told to always be helpful says “Yes, water damage is covered.” One told to say when it does not know replies, “I can’t confirm that. I can connect you with someone who can.”