Retrieval-augmented generation (RAG)
Definition
Retrieval-augmented generation (RAG) is a way of getting an AI model to answer from your own documents. A search step first finds the passages that match the question, then the model writes its answer using those passages.
By Ronan Liedmeier, fractional head of growth and course author · Published 17 October 2026 · Last updated 17 October 2026
Also called: RAG
You use it when your material is too large to paste into one prompt, when answers must point to a source, or when the documents change often. The name comes from a 2020 paper by Patrick Lewis and colleagues, which paired a language model with a retriever searching a vector index of Wikipedia.
How it works
- Documents are split into passages, a step called chunking.
- Each passage is turned into an embedding and stored in a vector database, ideally alongside a keyword index.
- When a question comes in, the system finds the passages closest to it, and a reranker can reorder them by relevance.
- The top passages go to the model with the question and an instruction to answer only from them and cite which passage each claim came from.
If the whole source fits in the model's context window, uploading it to a project can be simpler than building any of this.
Example
A customer support lead loads the company's help center articles into a RAG setup for her agents. An agent asks whether customers on the annual plan can pause billing. The system pulls the sections on pausing and on annual plans, and the model answers with links to both. When the policy changes, she edits the article, the system re-indexes it, and the next answer reflects the change.
Common mistake
Rewriting the prompt when answers come back wrong. Most bad RAG answers start with retrieval: the right passage was never fetched. Before touching the prompt, look at which passages the search returned for the failing questions. If they are wrong, fix the chunking or the search first.
Learn it in AI Academy
- RAG for One Document SetAdvanced · 16 minFree
- RAG: When It's the Right AnswerExpert · 9 min
Related terms
- Embedding
- An embedding is a list of numbers that represents the meaning of a piece of text, so a computer can measure how similar two texts are. Texts about the same idea get similar numbers even when they use different words.
- Vector database
- A vector database is a database built to store embeddings, the number lists that represent the meaning of text, and to quickly find the ones closest to a new question. In many RAG systems it is the place where the search step happens.
- Chunking
- Chunking is splitting long documents into smaller pieces before they go into a RAG system, so a search can find and return just the relevant part. Where you cut the text decides whether each piece still makes sense on its own.
- Context window
- A context window is the maximum amount of text an AI model can take into account at once, measured in tokens. It holds your messages, any files you add, its instructions and its own replies, and whatever doesn't fit is left out.