The big idea
RAG (retrieval-augmented generation) exists because your knowledge base is bigger than any context window, so instead of feeding the model everything, you find the relevant chunks first and let it answer only from those. Most RAG failures are retrieval failures, not model failures — the fix is almost always in what got fetched, not in the prompt.
Deep dive
The problem RAG solves
Context windows, even generous ones, cannot hold your entire knowledge base — years of proposals, a full policy manual, hundreds of client files. If you tried to paste all of it into one prompt, most of it wouldn't fit, and even if it did, the model would be searching for a needle in an enormous haystack of mostly irrelevant text.
RAG solves this by never asking the model to read everything at once. Instead, it searches your knowledge base for the specific pieces relevant to the current question, and hands the model only those pieces alongside the question. This is the entire reason RAG exists — it's a workaround for a very physical limitation, not a fancy feature for its own sake.
RAG in one sentence
Find the relevant chunks first, then let the model answer from them. That's the whole pipeline: a retrieval step searches your documents for content matching the question, and a generation step feeds those matched chunks to the model along with the original question, asking it to synthesize an answer grounded in what was retrieved.
Everything else — chunking strategy, embeddings, vector databases — is implementation detail in service of that one sentence. If you remember only this, you'll debug RAG systems correctly: always ask "what was retrieved?" before you ask "what did the model say?"
The honest alternative: when long-context is enough
Before building any RAG pipeline, check whether you actually need one. If your document set is small enough to fit inside a single upload — say, under a few hundred pages — a hosted assistant with Projects or simple file upload and long-context reading may already answer your questions correctly, with none of the chunking or retrieval complexity RAG introduces.
RAG earns its keep once your knowledge base is genuinely too large to upload wholesale, or once you need to search across a constantly-growing folder without re-uploading everything each time. Starting simple and only reaching for RAG when you hit a real wall keeps you from solving a problem you don't have yet.
Chunking: splitting documents into whole thoughts
Before anything can be retrieved, documents get split into chunks — smaller pieces that can be individually matched against a question. The goal is to split at natural boundaries so each chunk contains one whole thought (a full paragraph, a complete table, one policy clause) rather than cutting mid-sentence or mid-table.
Bad chunking is the single most common cause of RAG failures: a table split across two chunks means neither chunk has the full picture, so a question about that table gets a partial or wrong answer even though the information was technically in the document set. When retrieval keeps failing on a particular type of content, re-chunking that content is often the actual fix, not a better prompt.
Embeddings, intuitively
Embeddings turn text into a set of coordinates in a high-dimensional space, where meaning determines position: chunks about similar topics land near each other, and chunks about unrelated topics land far apart. When you ask a question, it also gets converted into coordinates, and retrieval simply finds the nearest chunks to that point.
You don't need the math to use this well — the intuitive picture is enough: similar meaning means nearby, and retrieval is a neighborhood search. What matters practically is that embeddings capture meaning, not just keyword overlap, so a question phrased differently than the document can still retrieve the right chunk, though not always perfectly.
Building it hosted
You don't need to build or manage a vector database to use RAG. Hosted options — an assistant with built-in file search, or a pre-built RAG template in a no/low-code automation platform — handle chunking, embedding, and retrieval behind the scenes. You upload a folder, and the system exposes a search or chat interface on top of it.
A consultant example: load 60 past proposals into a hosted file-search assistant, then ask "what did we quote for onboarding automation projects, and which arguments won?" The system retrieves the relevant proposal chunks, and the model synthesizes an answer with citations back to specific files — institutional memory, answering questions, without anyone building infrastructure.
Testing retrieval: the 10-question audit
Once your RAG system is live, don't just trust it — audit it. Write down 10 questions you already know the correct answer to, run each through the system, and for each one, inspect which chunks were actually retrieved before checking the final answer. This tells you whether failures are retrieval problems (wrong chunks fetched) or genuine model problems (right chunks, wrong synthesis) — the former is far more common.
In practice: two failures trace back to a table that got chunked badly, so those specific files get re-chunked, and a re-run scores 9 out of 10. That's the audit doing its job — you debugged retrieval, not the model, and you now have a concrete accuracy number instead of a vague sense that "it seems to work."
Knowing the limits
RAG has real edges worth knowing before you rely on it. Freshness: if your documents change, the retrieval index needs to be updated too, or you'll get confidently wrong answers from stale chunks. Tables and structured data: retrieval often struggles with information that's meaningfully tabular, since chunking can scramble rows and columns apart from their headers.
Multi-document synthesis is the hardest limit: if the true answer to a question spans five different documents, a system retrieving only the top few most-relevant chunks may miss pieces scattered across the set, producing an answer that's confidently incomplete. Knowing these limits up front means you can route those question types to a human or a different tool instead of trusting RAG blindly.
How to run it
- Golden set exists. A fixed set of 10-15 real questions with known-correct answers is saved and reused every time the RAG setup changes.
- Retrieved chunks are visible. You can see exactly which chunks were fed to the model for any given answer, not just the final response.
- "I don't know" is tested. At least some golden-set questions have no answer in the documents, and the system is confirmed to say so rather than hallucinate.
- Chunking respects structure. Tables, code blocks, and numbered lists are not split mid-unit by the chunking strategy.
- Failures are classified by layer. Every wrong answer is tagged as a retrieval failure or a generation failure before any fix is attempted.
In the field
A small law firm builds RAG over its 40-page employee handbook. The retrieval audit reveals that 4 of 15 test questions failed because the chunker split the PTO-accrual table across two chunks, so the model only ever saw half the table — a clean, fixable retrieval bug, not a prompting problem, found only because the audit separated retrieval failures from generation failures.
When not to use it
Don't reach for RAG when the whole document set comfortably fits in the model's context window with room to spare — for a single 10-page policy document, just upload it directly or paste it into a long-context prompt; the overhead of chunking, embedding, and retrieval infrastructure buys you nothing over simply including the whole thing. RAG earns its complexity only once the corpus is too large to fit, changes frequently enough that re-pasting is a hassle, or needs to be queried by many users without each one re-uploading it.
Pitfalls and takeaways
- Reaching for RAG before checking whether Projects or a long-context upload would already do the job
- Blaming the model for wrong answers when the real problem is bad retrieval
- Chunking documents so aggressively that whole thoughts (especially tables) get split apart
- Skipping a retrieval audit, so you never actually know your system's accuracy
- Assuming RAG can answer questions whose answer spans five different documents at once
- RAG exists because knowledge bases don't fit in context windows — retrieve first, then answer from what's retrieved
- Most wrong RAG answers are retrieval failures, not model failures — always inspect what was fetched first
- Check whether Projects/long-context already solves your problem before building a RAG pipeline at all
Do the work
Goal: Run a retrieval audit on your RAG setup to find out its actual accuracy, rather than assuming it works because a couple of test questions looked fine. (40 min)
- Write 15 realistic questions, not softballs. List 15 questions a real user would ask of this document set, including at least 3 that require combining info from two different chunks and 2 whose answer genuinely is not in the documents.
- Run each question and log what got retrieved. For each question, record not just the final answer but which chunks were actually retrieved and fed to the model — most RAG tools expose this via a "sources" or "citations" panel.
- Score each answer with a simple rubric. Mark each answer Correct, Partially Correct, Wrong, or Hallucinated (confidently wrong with no basis in the retrieved chunks). Tally the counts.
- Diagnose the failures by layer. For every non-Correct answer, decide: was the right chunk never retrieved (a retrieval failure) or was the right chunk retrieved but the model answered badly anyway (a generation failure)? Count each category separately.
- Fix the single biggest retrieval failure. Pick whichever failure mode is most common — bad chunking splitting a table, too few chunks retrieved (k too low), or a single document dominating results — and make one targeted fix.
- Re-run the full 15 and compare scores. Run the same 15 questions again after your fix and compare the before/after tally. Write one sentence on what improved and what's still broken.
Deliverable: A scored retrieval audit of 15 real questions (before and after one fix), with failures classified as retrieval vs. generation, saved as a reusable golden set.
Sources
- Anthropic docs & cookbooks (docs.claude.com, github.com/anthropics — RAG guides)
- OpenAI Cookbook (cookbook.openai.com — file search/RAG)
- Latent Space (latent.space) for practitioner context