Working With Long Context

A model with a very large context window can technically hold hundreds of pages of text at once, and that's genuinely useful — it means you can hand over an entire contract, codebase, or research corpus without pre-chunking it yourself. But there's an attention tax: everything in that context is competing for the model's limited attention, and a single relevant paragraph buried among 200 pages of noise gets diluted.

The capability of a large context window and the strategy of how you use it are two different things — having room for a million tokens doesn't mean the best move is filling it.

Research and practical experience both show that models tend to pay less attention to content placed in the middle of a long context than content at the beginning or end — a phenomenon called "lost in the middle." This means if the answer to your question happens to sit in the center of a huge pasted document, the model is measurably more likely to miss it or under-weight it compared to the same fact placed near the start or end.

This has a direct practical implication: placement matters, and pointing the model explicitly at where to look ("see section 4.2") compensates for attention that naturally degrades with distance from the edges.

The map-reduce pattern handles large documents more reliably than one giant paste: in the "map" phase, you extract structured information from each section or document individually ("liabilities, unusual terms, dates — structured"), then in the "reduce" phase, you run a synthesis pass over just those extractions ("patterns, contradictions, top risks") rather than over the raw source material again.

This is not just more reliable — it's also traceable: every fact in the final synthesis can be tracked back to the specific extraction, and from there to the specific source document, which matters enormously for due diligence, legal review, or any high-stakes reading task.

For long-running projects or documents that keep growing, the same map-reduce idea applies over time rather than over sections: periodically summarize the current state — decisions made, open questions, next steps — rather than letting the full history accumulate indefinitely in one context.

This progressive summarization keeps a living project navigable no matter how long it runs, because you're always working from a compact, current summary rather than re-reading the entire history each time you need to orient yourself.

When you know roughly where relevant information lives in a large document, say so explicitly rather than relying on the model to find it unaided: "using section 4.2 specifically, summarize the payment terms" dramatically outperforms a bare "summarize the payment terms" against the same large document.

Pointers cost you nothing extra to write and meaningfully improve accuracy, precisely because they counteract the lost-in-the-middle effect by directing attention rather than hoping the model self-navigates.

Long context and retrieval-augmented generation (RAG, covered in module A3) solve related but different problems. Long context is well suited to one-off, deep reads of a specific set of documents — read this whole data room once, thoroughly. RAG is better suited to repeated queries over a stable corpus that doesn't change often, because it retrieves only the relevant chunks each time rather than re-processing everything.

Choosing between them comes down to frequency and stability: a document you'll query once deeply favors long context; a corpus you'll query dozens of times over weeks favors RAG.

Long conversations degrade the same way long documents do — accumulated turns compete for attention and contradictions pile up. The fix is a checkpoint habit: roughly every 20 turns, ask for a summary of current state — decisions, open questions, next steps — and paste that summary into a fresh thread rather than continuing the increasingly cluttered original.

This single habit prevents the common failure mode of a 200-turn "sludge" thread that's technically still within the context window but has become functionally unreliable, and it costs almost nothing compared to the quality it preserves.