RAG — When It's the Right Answer

RAG earns its complexity when all three of these are true — if any one is missing, a simpler shape probably wins.

**1. The knowledge is too large or too volatile** to fit in context or bake into a fine-tune. A 300-page policy guide updated quarterly clears this bar. A 40-paragraph FAQ does not.

**2. Answers must come from your specific documents** — client SOPs, policy pages, product catalogues — not from the model's general training. If the model already knows the answer reliably, retrieval adds latency for no gain.

**3. You need traceable provenance.** 'This answer came from these three passages' is increasingly a hard requirement for regulated and brand-sensitive clients. If you can't point to a source, you're generating, not retrieving.

A chunk is the unit of retrieval — it's what gets pulled out of storage and handed to the model. Getting this wrong cascades into every downstream component.

**Fixed-size chunking** (~500 tokens, 10–15% overlap) is the robust default. It degrades gracefully across messy, inconsistently formatted documents.

**Semantic chunking** (splitting on headings and topic boundaries) pays off on well-structured docs like policy pages and API references. It tends to hurt on unstructured or messy source material.

**The rule that doesn't change:** a chunk read in isolation must still make sense. A chunk that opens with 'and therefore the limit is doubled' will retrieve fine and answer badly — there's no context for what doubled, or why.

Store generous metadata alongside each chunk — source URL, section title, effective date, client, access tier. It costs nothing at write time and enables filtering you'll definitely want later.

**Choosing an embedding model** comes down to three axes: retrieval quality on your specific domain (published leaderboards are directional, not decisive — measure on your own eval set), cost per million tokens (you embed the whole corpus, and if you switch models you re-embed everything — that migration cost is the real lock-in to think about), and dimensionality (storage and search latency scale with vector size).

Start with a solid mid-tier model, measure it against your eval set, and only move up when measurement says to.

**Retrieval strategy:** hybrid search — combining dense vector search with BM25 keyword search — is the default, not an optimisation. Pure vector search misses exact identifiers like SKUs and error codes. Pure keyword search misses paraphrase. You need both.

**Rerankers** sit on top of retrieval: they take the top ~50 candidates and reorder them, returning the best ~5. They are the highest ROI-per-line-of-code addition to a RAG stack. Add one before tuning anything else.

**Metadata filters** run before vector search and double as your access-control seam. Retrieval that ignores permission metadata is a data breach with extra steps.

This is an infrastructure decision with real tradeoffs — not just a performance one.

**pgvector** (Postgres-native) is the default for most Supabase deployments. Row-level security applies directly to chunks, your backup story is already sorted, and your team already knows Postgres. Use it until you have a measured reason not to.

**Managed vector stores** (Pinecone-style) trade ops burden for a new vendor dependency and data-residency questions that need to go on your client's risk register.

**Self-hosted engines** (Qdrant, Weaviate) give you full performance control — and full ownership of whatever breaks at 2 a.m.

For most client deployments: pgvector until you feel actual, measured pain.