Context rot

What is context rot?

Context rot is the gradual drop in how reliably a language model uses the text you give it as that text gets longer. The model still answers. It just gets less dependable at finding, combining and acting on any single fact buried in the pile.

The important part is the word gradual. This is not a cliff you hit at some token count, and it is not the same as running out of room. Performance slides as input grows, the slope differs from model to model, and it starts well before the context window is technically full.

Picture briefing a new colleague on a customer issue. Hand them one page with the account, the contract clause and the last two emails, and they act on it. Hand them the full 400-page archive and ask the same question, and they will often still find the answer, but the odds of them missing it, or quoting the wrong version of the clause, go up with every extra binder.

The name became common after Chroma published a report in July 2025 measuring the effect across eighteen models, and Anthropic picked the term up in its engineering writing on context.

Why performance drops as input grows

Two mechanisms sit underneath.

The first is attention. In a transformer, every token relates to every other token, so a prompt of n tokens produces roughly n squared relationships to weigh. Anthropic describes the result as an attention budget: the model has a finite pool of attention to spend, and every token you add takes some of it. Nothing breaks when the budget runs thin, the model just spreads itself over more material and gets vaguer about all of it.

The second is position. The 2023 paper Lost in the Middle tested models on multi-document question answering and key-value retrieval, moving the relevant document around inside the prompt and measuring accuracy at each position. Answers were most accurate when the needed information sat at the start or the end of the input, and noticeably worse when it sat in the middle, including in models built and marketed for long context. That is the lost in the middle effect, and it is the reason the position of a document in your prompt is a design decision, not an implementation detail.

What the testing actually shows

The best-known measurement is the needle in a haystack test, published by Greg Kamradt in late 2023. You hide one sentence at a chosen depth inside a long body of unrelated text, ask the model to repeat it back, and sweep two dimensions: how long the text is, and how deep the sentence is buried. Vendors quote high scores on it, and Google notes that Gemini does well on those retrieval evaluations while adding that accuracy is not the same once you look for several pieces of information rather than one.

That caveat matters, because the plain version of the test is easier than real work. It is a lexical match: the question repeats words from the planted sentence, so the model can find it by wording alone. Chroma's report ran a harder set of variations across eighteen models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3. A few results are worth carrying around:

• Performance changed significantly with input length even on tasks that are trivially easy at short length.

• When the question and the answer shared fewer words, and the model had to bridge the gap semantically, accuracy fell off faster as the input grew.

• Adding even one plausible-but-wrong passage cost accuracy, and some distractors did far more damage than others.

• Models scored better when the surrounding text was shuffled than when it read as a coherent document, which is the opposite of what most people assume.

• On a long-conversation benchmark, every model did better on a prompt trimmed to the relevant history than on the full transcript.

Chroma also ran a task with no retrieval in it at all: copy a list of repeated words back out. Accuracy still degraded as the list got longer, with models inventing words, undershooting the count, or declining the task. Length alone is enough to cause trouble.

Context rot versus a full context window

These two get mixed up, and the difference is what you see when things go wrong.

Hitting the window limit is loud. The API returns an error, or the oldest messages get dropped, and you know something was cut. You can fix it by chunking, summarising or retrieving less.

Context rot is quiet. Everything fits, nothing errors, and the answer comes back fluent and confident with the middle document quietly ignored. There is no log line for it. That is why teams often discover it through a user saying an answer was wrong rather than through monitoring, and why evaluating your assistant at the length it actually runs at is worth the effort.

What to watch out for with context rot

The pattern we see most often in projects is the same reflex: paste in the whole database schema, all the policy documents, the entire ticket history, and let the model sort it out. It is the fastest thing to build and the first thing to get flaky, usually right when someone asks a question that needs a fact from the middle.

Retrieve instead of dumping. Send the four tables that matter for the question rather than all two hundred. This is the job RAG exists to do, and a tight five thousand token prompt usually beats a five hundred thousand token one on accuracy, cost and latency at the same time.

Put the instruction where the model reads best. If you must send a long document, keep the question and the rules near the end rather than burying them behind it.

Prune near-duplicates. Three revisions of the same policy in the window is three chances to quote the outdated one. Deduplicate before retrieval results reach the prompt.

Test at real length. An assistant that answers perfectly against three sample documents tells you very little about its behaviour against three hundred. Build your evaluation set from questions people actually ask, and run it at production length.

Treat the window as a budget you spend, not space you fill. That reflex is the whole of context engineering, and context rot is the reason the discipline exists.