Chunking

The model sees only the chunks the search returns, not the documents they came from. A bad cut can hand it half a table or a rule without the exception that follows it.

How it works

The simplest method cuts text every so many words or tokens, often with some overlap between neighboring pieces. A better method for most business documents follows the structure: one chunk per section or subsection, using the headings, with tables kept whole.

Each chunk should also carry context: the document title, the section heading and a date, either inside the text or as metadata. Then test it. Run real questions, look at which chunks come back, and adjust the cuts until the right ones appear.

Example

A finance team puts its travel expense policy into a RAG assistant. Fixed-size cutting splits the per diem table, so one chunk holds the country names and the next holds the amounts. Questions about the daily rate in Germany come back wrong. They re-chunk by section heading, keep the table in one piece and repeat the heading at the top of each chunk. The answers come back right, and the prompt stays the same.

Common mistake

Cutting so small that chunks lose their meaning. A piece that begins "This limit applies only to contractors" can match a search well and still mislead, because it never says which limit. Keep the sentence that names the subject in the same chunk as the rule that follows it.