Long context versus RAG

What is long context versus RAG?

Long context versus RAG is a design decision you make once per AI feature: do you put the whole set of documents into the prompt, or do you build a retrieval step that looks up the relevant pieces per question and sends only those?

Until 2024 the question mostly answered itself, because the context window was too small to hold anything but a few pages. That changed. As of September 2026, every Claude model from the 4.6 generation onward takes a million tokens of input at standard pricing, and Google's Gemini Pro models have taken a million since 2.5 Pro. A dense page of text runs somewhere between 500 and 1,000 tokens, so a million tokens is on the order of a thousand to two thousand pages: a full policy manual, a year of board minutes, an entire supplier contract set, all in a single call.

That makes RAG optional where it used to be mandatory. It does not make it obsolete. The two approaches spend your money in different places: long context pays per question, RAG pays up front.

What long context does well, and what it costs you

Long context shines when the set of documents is one thing, or a small and stable collection, and the questions need all of it at once. Read this contract and list every clause that shifts liability to us. Compare these three tenders on delivery terms. Summarise the whole audit report. A retrieval step would have to guess which passages matter, and for questions like these the honest answer is all of them.

The second advantage is that there is nothing to build. No pipeline, no index, no embedding step. You load the documents, you ask, you get an answer, and a first working version costs a fraction of the effort a retrieval pipeline does.

The costs come in three forms. The first is tokens per call. Every question resends the full set, so a 300-page set costs the same couple of hundred thousand input tokens whether the question needs one clause or all of them. The second is latency: Google's Gemini documentation says plainly that longer prompts take longer before the first token comes back. The third is accuracy, and it is the one people forget. A model does not use a full window as well as it uses a short one. The context rot entry covers the measurements; the short version is that recall gets worse as input grows, that facts in the middle of a long prompt are found less reliably than facts at the start or end, and that a prompt trimmed to what matters usually beats the full one.

The mitigation for the token cost is prompt caching. If the document set sits at the front of the prompt and does not change between questions, the provider stores the processed prefix and charges a fraction for reusing it: on Anthropic's API a cache read costs a tenth of the normal input price, and a fortieth on the newest models. Anthropic's own docs use exactly this case as the example, a 50-page legal agreement cached in the system message and queried repeatedly. Caching does nothing for accuracy, and it needs the next question to arrive within the cache lifetime, five minutes by default.

What RAG does well, and what it costs you

RAG handles what no window can. A corpus of 20,000 documents runs to a hundred million tokens or more, and no amount of context growth changes the fact that a question about one invoice should not read every invoice.

Beyond size, four things only RAG gives you. Access control per document, because retrieval can filter on who is asking, and the sales rep then never sees the HR file. Freshness, because a document indexed at nine is answerable at five past nine without anyone rebuilding a prompt. Citations, because the model saw specific passages and can point at them. And cost per question: twenty retrieved chunks are a few thousand tokens rather than a few hundred thousand, at any corpus size.

The price is a pipeline. You have to parse the documents, choose a chunking strategy, run an embedding model, keep a vector database or search index up to date, and keep the permissions in the index in sync with the permissions in the source system. That is software, and software has to be maintained.

The other price is retrieval failure. If the right passage is not in the retrieved set, the model answers from whatever it did get, fluently and wrong. Anthropic measured this in 2024 on its own test corpora: a plain embedding search missed the needed chunk in the top twenty results about six percent of the time, and adding contextual descriptions and a reranker brought that down to about two percent. Two percent is good. It is also a failure mode that long context simply does not have.

Long context versus RAG: what you pay per question and what you build once

The cleanest way to compare the two is to separate the recurring cost from the one-off cost.

Long context has a one-off cost near zero and a recurring cost proportional to the size of the set. Every question carries the whole set, at full price without caching and at a tenth or less with it. The accuracy risk also grows with the set, because more tokens in the window means more context rot.

RAG has a real one-off cost, the pipeline and the index, plus a maintenance cost that scales with how often the documents change. The recurring cost per question is small and does not grow with the corpus. The accuracy risk sits in retrieval rather than in the window: the model reads a short, relevant prompt, but only if retrieval found the right passages.

A 2024 study by a team at Google DeepMind and the University of Michigan compared both on the same questions and landed on one sentence: given enough context, long context scored higher on average, and RAG's much lower cost stayed its distinct advantage. The gap ran from roughly 4 to 13 percentage points depending on the model. So you are buying accuracy on cross-document questions with tokens, or buying cheap questions with build effort and a retrieval risk.

The hybrid most production systems land on

In practice, the systems we see running well do not pick a side. They retrieve generously and then give the model a large but bounded context.

The pattern is: use search to narrow the corpus to the fifty or hundred passages, or the five or ten documents, that could plausibly matter, then hand all of that to a model with a big window rather than the three chunks that RAG systems used to send when windows were small. Retrieval does the job it is good at, cutting millions of tokens down to tens of thousands, and the window does the job it is good at, reading across the survivors and combining them.

The same DeepMind team proposed a routing version of this: try the retrieved passages first, let the model say whether they are enough, and only fall back to the full context when they are not. On their numbers that kept accuracy close to full long context at 40 to 65 percent lower cost, depending on the model. Anthropic describes the same shape in its engineering guidance for agents: a few reference files loaded into context up front, everything else fetched just in time through search tools. Claude Code works that way, with its instruction files always present and the codebase reached through grep.

Bounded is the word that does the work. Databricks ran that test across thirteen models in 2024 and found answer quality improving as more retrieved context was added, up to a point, and then falling off. The turning point moved with the model and the document set, from a few thousand tokens to well past a hundred thousand. So the hybrid is not retrieve everything and dump it. It is retrieve a generous set, then measure where your model stops getting better.

A decision rule you can use in a meeting

Rounded numbers, meant to be argued with rather than applied blindly.

• Under a few hundred pages, stable set, everyone allowed to see all of it: long context with prompt caching. Anthropic's own guidance since 2024 has been that a knowledge base under roughly 200,000 tokens, a few hundred dense pages, can simply go in the prompt. Skip the pipeline.

• More than that, or per-user permissions, or documents that change daily: RAG. Any one of those three is enough. Permissions alone rule out a shared prompt, and a set that changes every day breaks the cache anyway.

• Questions that span many documents at once: both. Retrieve a generous set, then give the model tens of thousands of tokens rather than a handful of chunks.

A worked example from the kind of company we usually work with.

A manufacturer has forty pages of supplier contracts, five documents, renegotiated once a year. Purchasing asks perhaps two hundred questions a month: notice periods, price revision clauses, who carries transport risk. Forty dense pages is around 40,000 tokens. On Claude Sonnet 5 at 2 dollars per million input tokens that is about 8 cents a question, so some 16 dollars a month, and with caching under a cent a question. The whole feature is a system prompt with five PDFs in it. A retrieval pipeline for this would cost more than a decade of questions.

The same company's SharePoint holds 20,000 documents: quotes, delivery notes, quality reports, HR files. At a few thousand tokens each that is easily a hundred million tokens, a hundred times any window, and half of it is not meant for everyone. This is RAG, with the index filtering on the asker's permissions. Each question sends maybe twenty retrieved chunks of 500 tokens, about 10,000 tokens, or 2 cents on the same model. The one-off cost is the pipeline: parsing, chunking, an index, and a sync job that follows document changes and permission changes.

And when quality asks which suppliers had a delivery complaint in the last quarter that also had a price revision in their contract, that is a cross-document question. Retrieve the complaints and the contract clauses, then let a large window read them together.

What to watch out for with long context versus RAG

The window is a limit, not a target. Google reports near-perfect scores when a model has to find one planted fact in a long prompt, and adds that accuracy drops once the question needs several facts at once. Real questions need several facts. Test at the length you will actually run.

A change in the set breaks the cache. One edited document at the front of the prompt means the next call pays full price for the whole set. If the documents change more often than the cache lifetime, you are not caching, you are paying the write premium on every call.

Pricing tiers move at 200,000 tokens on some providers. Anthropic bills the full million at one rate. Google charges double per input token on Gemini Pro above 200,000 tokens. Check the pricing page for the model you actually use before you size the set.

RAG does not give you permissions for free. Filtering the index on the asker's identity only works if the index knows the permissions, and the sync job that keeps it right is the part most projects underbuild.

Measure both before you commit. Take fifty real questions, run them against the full set in one prompt and against a retrieval prototype, and compare the answers. The rule above is a starting point; your documents and your model decide.