Image: Anthropic

Anyone who has spent an hour in a single chat with an AI assistant has probably seen it happen. An instruction given at the start, such as keeping answers short or avoiding a certain library in code, slowly stops being followed. A detail shared twenty messages ago gets ignored or mixed up with something else. The chatbot has not crashed, and the earlier message is still sitting in the conversation. It simply seems to have lost track of it.

This can feel strange, because AI companies now advertise very large context windows, with some models able to take in hundreds of thousands or even millions of tokens at once. If the whole conversation fits, why does the model behave as if parts of it are missing? A growing body of research gives a fairly clear answer. Fitting text into a model is not the same as the model using all of it equally well.

A context window is the amount of text a language model can consider at one time. That text is measured in tokens, which are small chunks of words, word pieces or punctuation. Everything the model works with in a chat counts toward this limit, including the system instructions set by the company, the full back and forth of the conversation, any pasted documents and the results of any tools the assistant uses.

Anthropic describes context as "the set of tokens included when sampling from a large-language model" in an engineering post on context engineering published in September 2025. The company explains that models are built on the transformer architecture, where every token can attend to every other token. That creates a pairwise relationship between every two tokens, so the number of relationships grows much faster than the length of the text. According to Anthropic, as context grows, a model's ability to capture these relationships "gets stretched thin."

Anthropic also points to training. Models learn their attention patterns from data where shorter sequences are more common than longer ones, so they have less experience with dependencies that span a whole long context. The company calls the result an "attention budget" and says every new token depletes it by some amount. In its words, this creates "a performance gradient rather than a hard cliff." A model does not suddenly fail when a chat gets long. It gradually becomes less precise.

One of the best known studies on this problem is Lost in the Middle, a paper by researchers from Stanford University, the University of California, Berkeley and Samaya AI. The team tested how well models found and used a relevant piece of information when it was placed at different points in a long input. In one task, models answered questions using a set of documents where only one contained the answer. In another, they had to pull a single value out of a long list of key and value pairs.

The researchers found what they called a U-shaped performance curve. Models did best when the relevant information sat at the very beginning or the very end of the input, and performance dropped significantly when it sat in the middle. In one example from the paper, GPT-3.5-Turbo's accuracy on the multi-document task fell by more than 20 percent in the worst case. With 20 or 30 documents, its worst-case score was lower than the 56.1 percent it scored when given no documents at all and asked to answer from memory.

The same paper also tested models sold with extended context, such as versions of GPT-3.5-Turbo and Claude-1.3 with larger windows. When the input fit inside both the regular and extended versions, their results were nearly identical. The authors concluded that extended-context models are not necessarily better at using their input. A bigger window lets more text in, but it does not guarantee the model will pay attention to all of it.

Newer work suggests the problem has not gone away with newer models. In July 2025, the vector database company Chroma published a technical report titled Context Rot. The team evaluated 18 models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3. According to the report, models do not use their context uniformly, and "their performance grows increasingly unreliable as input length grows."

Chroma deliberately kept its tasks simple so that the only thing changing was the length of the input. Even so, it reported that performance consistently degraded as inputs got longer across all of its experiments. Adding even one distractor, a sentence that looks related to the question but does not answer it, reduced performance compared to a clean baseline, and four distractors made it worse. In a test based on a conversational memory benchmark called LongMemEval, models scored significantly higher on focused prompts of roughly 300 tokens than on full prompts of roughly 113,000 tokens that also contained irrelevant chat history.

A separate benchmark called NoLiMa, from researchers at LMU Munich and Adobe Research, pushed on a related weakness. Many long context tests let models find an answer by matching the exact words in the question. NoLiMa removes that shortcut. In one example from the paper, a hidden sentence says a character lives next to the Semper Opera House, and the question asks which character has been to Dresden. The model has to make the connection on its own.

The paper tested 13 models that claim to support at least 128,000 tokens. They performed well on short inputs under 1,000 tokens, but performance dropped significantly as the input grew. At 32,000 tokens, 11 of the 13 models fell below half of their short context scores. GPT-4o, one of the strongest performers, went from 99.3 percent to 69.7 percent. The authors said the declines appear to come from the attention mechanism struggling in longer contexts when there are no literal word matches to lean on.

Long documents are one part of the story. Long conversations add another problem, because the information arrives in pieces over many turns. Researchers from Microsoft Research and Salesforce Research studied this in a paper titled LLMs Get Lost In Multi-Turn Conversation.

Image: Laban et al., Microsoft Research and Salesforce Research

The team took tasks such as writing code, writing database queries, solving math problems and summarizing documents, and split each instruction into smaller pieces revealed one turn at a time. They then compared this with giving the model the full instruction at once. Across more than 200,000 simulated conversations with 15 models, every model they tested did worse in the multi-turn version, with an average drop of 39 percent across six tasks.

The authors found that most of this drop did not come from models becoming less capable. Their best-case ability fell by an average of 16 percent, while unreliability, the gap between their best and worst runs, rose by 112 percent. In plain terms, the same model on the same task became much more unpredictable once the instructions were spread across a conversation. The paper says models often make assumptions in early turns, try to give a final answer too soon and then rely too heavily on those earlier attempts. As the authors put it, when models take a wrong turn in a conversation, they get lost and do not recover.

The paper also tested a simple fix. In a setup it called Snowball, each new turn repeated everything the user had said so far. That reduced the drop in performance by 15 to 20 percent, according to the authors, but did not close the gap. Lowering the model's temperature, a setting that controls randomness, also did little to make multi-turn conversations more reliable in their tests.

AI companies are aware of these limits, and much of the response so far has focused on managing what goes into the context window rather than only making it bigger. Anthropic calls this context engineering and describes the goal as finding "the smallest possible set of high-signal tokens" that make the desired result most likely. The company says waiting for larger windows is unlikely to solve the problem, since windows of all sizes will still face what it calls context pollution.

One technique Anthropic describes is compaction. When a conversation nears the limit, the model summarizes it and a new context window starts from that summary. In Claude Code, Anthropic says the summary keeps details such as architectural decisions and unresolved bugs while discarding redundant tool output. The company warns that overly aggressive compaction can lose subtle details whose importance only shows up later.

Other approaches covered in the post include structured note-taking, where an agent writes notes to a memory outside the context window and reads them back later, and sub-agents that each work with a clean context and return only a short summary. Anthropic also describes a just in time approach to retrieval, where an agent keeps lightweight references such as file paths or links and loads the actual data only when it needs it. The Lost in the Middle authors made a similar point about retrieval in their own work, noting that adding more retrieved documents beyond 20 gave only small gains for the models they tested.

For everyday users, the research points to a few habits worth trying, and these are recommendations drawn from the findings rather than rules from any one company. Put everything important in the first message. A clear, complete request at the start avoids the drift the multi-turn study measured, since that paper found providing all the information at once was the only reliable way to avoid the drop.

Start a fresh chat when a conversation goes off track. Arguing with a confused assistant over many more turns tends to pile on more context for it to sort through. The Microsoft and Salesforce authors suggest asking the model to consolidate everything said so far, then pasting that summary into a new conversation.

Keep pasted material relevant. The Chroma results suggest that unrelated text and near miss distractors can hurt answers even when the right information is present, so trimming long documents down to the parts that matter is likely to help. When a key rule matters, repeat it near the end of a long prompt or restate it later in the chat, since the research points to the beginning and the end of the input as the places models handle best.

Large context windows are a real improvement, and they let chatbots take in far more material than before. But the research so far shows that capacity and attention are two different things. A model can hold a whole conversation and still give uneven weight to parts of it, especially the middle and the details that were never repeated. No major AI company claims to have fully solved the problem at the time of this writing, and the practical fixes still come down to careful management of what the model sees. For more on how chatbots fall short, see earlier explainers on why AI chatbots still make things up and why they tell you what you want to hear.