Definition
Retrieval-augmented generation (RAG) is a method where a system first retrieves relevant passages from a document collection, then gives them to an LLM along with the question so the answer is based on that text.
How retrieval-augmented generation works
The term comes from a 2020 paper by Lewis and colleagues. It combined a pre-trained sequence-to-sequence model with a searchable index of Wikipedia passages. A neural retriever fetched passages relevant to the input, and the generator used them to produce its output. The idea is older and broader than that one design, and the name now covers many systems.
A typical version has four steps.
- Index. Documents are split into passages and stored in a searchable index, often as numeric embeddings so that search works on meaning as well as on words.
- Retrieve. The user’s question is turned into a query, and the closest passages are returned.
- Augment. Those passages are placed into the LLM’s input next to the question and the instructions.
- Generate. The model writes an answer, ideally using only what the passages say.
The benefit is that you can change what the system knows by changing the documents, without retraining a model. A knowledge base is the document collection such a system searches.
Retrieval does not guarantee a correct answer. The search can return the wrong passage or none, the passages can contradict each other, and the model can still add things that are not in them. Retrieval reduces hallucination and does not remove it.
Why it matters for voice agents
On a call, retrieval happens while the caller waits. The search and the longer input both add to voice latency, so teams keep passages short, limit how many they retrieve, and sometimes search only when the question needs it.
The query also starts as a speech transcript. A misheard product name or a half-finished sentence can send the search to the wrong place. Rewriting the question before search, and giving the recognizer a list of expected terms, both help.
The output is heard and not read. A caller cannot see a list of sources or a long excerpt, so the agent has to turn retrieved text into one or two spoken sentences and say plainly when nothing relevant was found.
Example
An illustrative retrieval for a clinic line:
- Caller: “Do you take walk-ins on Saturday?”
- Retrieved passage: “Saturday hours are 9 to 1, by appointment only.”
- Agent: “Saturdays are by appointment only, from 9 to 1. Would you like to book one?”