No, long context did not kill RAG
Every time a model ships a bigger window, someone declares retrieval obsolete. The economics and the recall curves both say otherwise — but the architecture is genuinely changing.
The argument goes like this: if the window holds two million tokens, just put the documents in it. No chunking, no embeddings, no vector database, no reranking. The whole retrieval stack collapses into a file read.
It's a clean argument. It's also wrong in three specific ways, and being precise about which ways matters, because one part of it is genuinely right.
The economics don't work
Consider a support assistant over a 500-page product manual — roughly 350K tokens. A retrieval pipeline pulls maybe 4K tokens of relevant context per query. Stuffing the full manual costs about 85× more per query.
At a hundred queries a day, the difference is noise; you should absolutely skip the retrieval stack and keep your life simple. At a hundred thousand queries a day, it's the difference between a viable product and one that loses money on every interaction.
The rule of thumb: long context wins on low volume over small corpora. Retrieval wins as either number grows. Most people arguing about this are arguing from different points on that curve without saying so.
Recall degrades with depth, and it isn't uniform
The more interesting objection is that a large window doesn't mean uniform attention across it. Plant a fact at varying depths in a long context and measure whether the model recalls it, and you get a curve, not a flat line — strong at the start, strong at the end, softer through the middle.
This has a practical consequence people skip: a model can hold a document without being able to reliably use it. Context length is a capacity claim, not a recall guarantee. If your eval only tests facts near the edges, you'll ship something that fails on the middle and looks like a reasoning problem.
Latency is a product decision
Prefill scales with input length. A 300K-token context has a time-to-first-token measured in seconds, not milliseconds — before generation starts.
For a batch summarization job, irrelevant. For anything a human waits on, it's the difference between a tool people use and one they route around. Caching helps enormously when the prefix is stable, and not at all when each query brings different documents.
Where the skeptics are right
Naive RAG — fixed-size chunks, single-vector similarity, top-k concatenation — is genuinely obsolete, and defending it is what makes the "RAG is dead" crowd sound correct.
What replaces it isn't no-retrieval. It's better retrieval:
- Larger chunks. When the window was 4K, 512-token chunks made sense. At 200K, retrieve whole sections and let the model do the narrowing. The chunk-size anxiety of 2023 was an artifact of scarcity.
- Retrieve then rerank. Cheap wide recall, then a cross-encoder to cut candidates down. One extra call, dramatically fewer distractors.
- Hybrid always. Dense vectors miss exact identifiers, error codes, and part numbers. BM25 catches them. Running both and fusing is close to free.
- Agentic retrieval. Let the model issue its own queries, look at what came back, and search again. Multi-hop questions stop being a special case.
What to actually do
| Situation | Approach |
|---|---|
| Corpus fits in window, low volume | Skip retrieval, stuff the context |
| Corpus fits, high volume | Retrieve — cost dominates |
| Corpus exceeds window | Retrieve, no choice |
| Latency-sensitive | Retrieve, keep prefill small |
| Exact-match matters | Hybrid, always |
The framing that survives is not "retrieval versus long context." It's that both are mechanisms for deciding what the model looks at, and the decision has moved from an architectural constant to a per-query one. That is a more interesting problem than the one people are arguing about.