RAG: When It's the Right Answer

Lesson preview

A wrong answer with correct chunks in context is a generation problem. A wrong answer with wrong chunks is a retrieval problem — and no amount of prompt surgery fixes it. Always test retrieval alone, first. RAG earns its complexity when all three of these are true — if any one is missing, a simpler shape probably wins. 1. The knowledge is too large or too volatile to fit in context or bake into a fine-tune. A 300-page policy guide updated quarterly clears this bar. A 40-paragraph FAQ does not. 2. Answers must come from your specific documents — client SOPs, policy pages, product catalogues — not from the model's general training. If the model already knows the answer reliably, retrieval adds latency for no gain. 3. You need traceable provenance. 'This answer came from these three passages' is increasingly a hard requirement for regulated and brand-sensitive clients. If you can't point to a source, you're generating, not retrieving. A chunk is the unit of retrieval — it's what gets pulled out of storage and handed to the model. Getting this wrong cascades into every downstream component. Fixed-size chunking (~500 tokens, 10–15% overlap) is the robust default. It degrades gracefully across messy, inconsistently formatted documents. Semantic chunking (splitting on headings and topic boundaries) pays off on well-structured docs like policy pages and API references. It tends to hurt on unstructured or messy source material. The rule that doesn't change: a chunk read in isolation must still make sense. A chunk that opens with 'and therefore the limit is doubled' will retrieve fine and answer badly — there's no context for what doubled, or why. Store generous metadata alongside each chunk — source URL, section title, effective date, client, access tier.

Key terms in this module

Chunking

Chunking is splitting long documents into smaller pieces before they go into a RAG system, so a search can find and return just the relevant part. Where you cut the text decides whether each piece still makes sense on its own.

Embedding

An embedding is a list of numbers that represents the meaning of a piece of text, so a computer can measure how similar two texts are. Texts about the same idea get similar numbers even when they use different words.

Fine-tuning

Fine-tuning is extra training that adjusts an existing AI model on your own examples so it learns a specific style, format or task. It changes the model itself, while a prompt or a document search only changes what the model sees for one request.

Retrieval-augmented generation (RAG)

Retrieval-augmented generation (RAG) is a way of getting an AI model to answer from your own documents. A search step first finds the passages that match the question, then the model writes its answer using those passages.

Semantic search

Semantic search is a way of searching by meaning instead of exact words, so it finds passages that say what you asked even when they use different wording. It usually works by comparing embeddings, which are lists of numbers that capture what a piece of text means.

Vector database

A vector database is a database built to store embeddings, the number lists that represent the meaning of text, and to quickly find the ones closest to a new question. In many RAG systems it is the place where the search step happens.

Frequently asked questions

Who is "RAG: When It's the Right Answer" for?

This Expert module in AI Delivery Skills & Workflows is for operators, founders and growth teams who run AI in real work and want production-grade judgement. It takes about 9 minutes and ends with a 5-question quiz.

Do I need to code?

Not to follow the lesson. The concepts are explained in plain language; if you build AI systems yourself, the exercise shows where the ideas map onto code.

What comes before and after this module?

Before "RAG: When It's the Right Answer", take "The Agentic Build Pattern" (module 1 of AI Delivery Skills & Workflows). Afterwards, continue with "The Eval Workflow".

Related modules

• Previous
  The Agentic Build Pattern

  An agent is a control loop, not a personality.

• Next
  The Eval Workflow

  Error analysis → judges → regression in CI. The skill clients cannot get from a demo.

• Same topic
  RAG for One Document Set

  Make AI answer from YOUR documents — one folder, one retriever, no vector-DB theology.

• Same topic
  The Adoption Playbook

  OpenAI × Anthropic — the frontier labs' interlocking evidence.