Context Windows Got Huge. Here's Why Your Agent Still Forgets
- Long context windows limitations are real: advertised token counts (1M and up) are marketing ceilings, not the point where models still recall reliably.
- On the NoLiMa benchmark, 11 of 12 models fell below half their short-context accuracy by just 32K tokens, long before their limit.
- The "lost in the middle" effect persists: models read the front and back of a prompt well and quietly drop what sits in the center.
- Retrieval, memory design, and context engineering still do the heavy lifting. A bigger window changes the budget, not the behavior.
- Build agents around what the model actually does with context, not around the number on the spec sheet.
Every few months a new model ships with a context window that would have sounded absurd in 2023. A million tokens. Ten million. The pitch writes itself: stop fussing with retrieval, just paste the whole knowledge base in and let the model sort it out. I have worked with these systems since 2016, and I will tell you where that pitch quietly breaks. The long context windows limitations that matter are not about how much you can stuff in. They are about how much the model still uses well once it is in there.
The Promise of Infinite Context
The promise is that scale ends the memory problem. If the window is big enough to hold your documents, your chat history, and your instructions all at once, then the agent should never forget anything you told it. That is the story behind the token-count arms race, where vendors advertise windows from roughly 1,048,576 tokens on Google's Gemini up to 10 million on Meta's Llama 4 Scout.
It is a seductive story because it is half true. You genuinely can fit more in. The trouble is that a context window is working memory for one request, not a place the model stores anything. As IBM Research puts it, once you exceed the window earlier content is truncated rather than remembered. Fitting more in and recalling it reliably are two different problems, and only the first one got solved.
The Real Limitations of Long Context Windows
Here is the part the spec sheet leaves out. Effective recall collapses well before you reach the advertised maximum. On NoLiMa, a benchmark that strips out the literal keyword overlap models lean on, 11 of 12 tested models dropped below 50% of their short-context baseline once the input hit just 32K tokens. That is a fraction of a million-token window, and performance had already halved.
NVIDIA's RULER benchmark found the same shape from a different angle: a model's effective context length, the point where it still beats a strong short-context baseline, sits far below its claimed length. Several models advertised at 32K and beyond failed before they even reached that number. When someone tells you the window is a million tokens, the honest follow-up question is: reliable through how many of them?
'Lost in the Middle' Hasn't Gone Away
The oldest of these problems is also the most stubborn. The Lost in the Middle study showed that models use information best when it appears at the very start or end of a prompt, and get noticeably worse when the key fact is buried in the center. In the authors' own words, performance "significantly degrades when models must access relevant information in the middle of long contexts."
That was 2023, and it has not aged away. Chroma's Context Rot report tested 18 frontier models and found every one degraded as input length grew, even on simple tasks where the answer was sitting right there in the prompt. They named the effect well: context rot. The window does not fail all at once. It gets soft in the middle, and your agent's "forgotten" instruction is usually a detail that landed there.
Where Retrieval and Memory Earn Their Keep
This is why retrieval did not become obsolete the moment windows got large. The job of retrieval is to put the few things that matter near the model's attention instead of drowning them in everything that does not. I wrote a plainer walkthrough of that mechanic in RAG in Plain English, but the short version is: relevance beats volume.
The choice between the two is not religious. Use a big context window when the material is already small enough to fit and needs to be reasoned over together in one pass. Reach for retrieval when the knowledge base is large, changing, or mostly irrelevant to any given question. Long context does not replace retrieval, because every irrelevant token you add is noise the model has to see past, and past a point it stops seeing past it. Getting your own documents into a shape an agent can actually pull from is its own discipline, one I covered in turning your documents into a brain an agent can use.
Designing Around Long Context Window Limitations
So what do you actually do about it? Design for how the model behaves, not for the number in the pricing table. Put the instruction you most need obeyed at the start or the end of the prompt, never in the middle. Keep the working context lean, and let retrieval feed it the relevant slice rather than the whole corpus. Give the agent real memory (a store it writes to and reads from) instead of pretending the window is a filing cabinet.
None of this is exotic. It is the same lesson as metadata and clean data: the boring plumbing decides whether the smart model looks smart. A bigger window is a bigger table to work on. It does not do the organizing for you. The teams whose agents remember are the ones who kept designing the memory even after the window got huge.
Frequently Asked Questions
Do Big Context Windows Solve AI Memory?
No. A larger window gives the model more working memory for a single request, but it creates no memory that persists across sessions, and reliability still degrades as the input grows. Durable memory needs a store the agent reads from and writes to.
What Is 'Lost in the Middle' in LLMs?
It is the tendency of models to use information best when it sits near the beginning or end of a prompt and least reliably when the key fact is buried in the middle of a long context. It was documented in 2023 and still shows up in current long-context testing.
RAG vs Long Context: When Should You Use Each?
Use retrieval to pull only the most relevant facts from a large or changing knowledge base. Use long context when the material is already small enough to fit and must be reasoned over together. Long context is not a substitute for retrieval, because irrelevant tokens add noise that lowers accuracy.
What Is Context Rot?
Context rot is the decline in output quality as a prompt gets longer, especially when important details are deep in the context, rather than a hard cutoff at the token limit. Chroma found it across 18 frontier models even when the answer was present in the prompt.
Is More Context Always Better?
No. Extra tokens help only when they are relevant. Otherwise they add noise, latency, and cost, and they raise the chance the model misses the evidence that actually mattered.