Short answer: Models usually use the start and end of a long input more reliably than the middle, so important details buried mid-context often get ignored.
A larger window does not treat every token equally. Attention skews toward the edges, so stuffing more history into the middle can still fail. Memory systems exist so important facts do not depend on where they sit in a long prompt.
Suppose the size problem is set aside for a moment. Suppose a model’s context window is genuinely large enough to hold everything that matters for a task, without needing to truncate anything or resend a growing transcript at ruinous cost. Even then, cramming more into that space doesn’t behave the way most people expect. Where a piece of information sits inside a long input measurably changes whether the model actually uses it, and the pattern isn’t random: content near the very start or the very end of a long context tends to get used well, while content buried in the middle is often functionally ignored, even though it was technically right there the whole time. This matters because it means a context window isn’t a neutral container that treats everything inside it equally. It has a shape to its attention, and understanding that shape explains why simply feeding a model more history, more documents, or more retrieved chunks can make a response worse rather than better.
What Does It Actually Mean for a Model to Get “Lost in the Middle”?
The phrase describes a specific, measurable effect that researchers found when testing how well models could locate and use a piece of information depending on where that information appeared within a long input. The setup is simple: place the answer to a question somewhere inside a long passage, run the same question with the answer positioned at different points, near the beginning, in the middle, near the end, and measure how often the model gets it right. The result wasn’t a smooth decline as the passage got longer. It was a distinct dip specifically for content in the middle, forming something closer to a U-shaped curve: strong performance at both ends of the input, and a meaningful drop for whatever sits in between.
What makes this worth taking seriously rather than treating as a minor quirk is that the information wasn’t hidden or ambiguous. It was present, complete, and directly relevant to the question being asked. The model simply attended to it less than it attended to content near the edges of its input. For an agent pulling together several retrieved documents, a long tool output, and a running conversation history into a single request, that means the order in which pieces of information get arranged, not just whether they’re included at all, can determine whether the model actually notices the one that matters.
Why Does Position Affect Attention Like This at All?
The honest answer is that this isn’t something anyone deliberately designed into these models; it’s a byproduct of how they were trained and how they process sequences. During training, models see enormous numbers of examples, and certain positional patterns, like the beginning of a document often containing a summary or the most salient framing, and the end often containing a conclusion or the most recent, most immediately relevant turn, end up correlating with usefulness far more often than the middle does. Over time, that correlation shapes how attention gets allocated by default, even when a specific input doesn’t follow that pattern at all.
There’s also a more structural piece to it. As an input sequence grows, every token in it is, in principle, competing for a limited amount of attention capacity spread across the whole sequence. Content at the edges benefits from proximity to the parts of the sequence the model was trained to weigh most heavily, the initial framing and the most recent input, while content stuck in the middle has to compete against everything else for attention that isn’t specifically anchored to it. The practical result is what researchers describe as effective context length, meaning the portion of a stated context window a model can actually use reliably, falling well short of the full advertised size. A model rated for hundreds of thousands of tokens might only reliably use a fraction of that before its accuracy on buried details starts to slide.
What Does This Mean for an Agent Pulling Together Multiple Sources at Once?
This effect stops being an academic curiosity the moment an agent is doing anything more complex than answering one question about one short passage. Real agentic tasks routinely combine a system prompt, several retrieved documents or memories, some tool output, and a running conversation history into a single request. If the one detail that actually answers the user’s question happens to land in the middle of that combined block, purely because of how the pieces were concatenated, the model may simply fail to use it, not because retrieval found the wrong thing, but because the right thing got buried by position rather than content.
This creates a subtle trap for systems that assume “if it’s retrieved, it’s available.” Retrieval can work perfectly, finding exactly the right memory or document, and the response can still come out wrong or incomplete because the position that content ended up in after everything got stitched together happened to be the position the model pays the least attention to. It also means that simply retrieving more results, on the theory that more relevant context can only help, can backfire: padding a request with additional retrieved chunks pushes the position of the genuinely important one further from the edges and deeper into the part of the window that gets the least reliable attention.
Doesn’t a Bigger or Newer Model Just Fix This Over Time?
It’s reasonable to expect that as models improve, this effect might simply go away, and it’s true that newer models have gotten somewhat better at handling long inputs than earlier generations. But “somewhat better” is doing a lot of work in that sentence. Effective context length, the portion a model reliably reasons over, continues to trail well behind the maximum stated context length across generations of models, even as that maximum keeps growing. A model advertising a much larger window than its predecessor doesn’t automatically mean the middle of that larger window is any less prone to being underused; it often just means the effect shows up further out, at a scale most single requests never even reach.
This is worth being direct about, because it undercuts a specific, common assumption: that the fix for weak agent memory is simply waiting for context windows to keep growing, or switching to whichever model currently claims the largest one. Position-dependent attention is a property of how these models process sequences, not a temporary limitation of any one model generation, and there’s no guarantee it disappears just because the number printed on a model’s context length spec gets bigger. Treating a growing context window as a substitute for a properly designed memory system means inheriting this weakness indefinitely, no matter which model happens to be running underneath.
How Does Weaviate Engram Avoid Depending on Position at All?
The most direct way to sidestep the lost-in-the-middle effect isn’t to get better at arranging what goes into a long context. It’s to avoid needing a long context full of mostly irrelevant material in the first place, so that whatever does get sent to the model is already small and genuinely relevant, rather than one useful fact diluted among dozens of tokens of history that happen to surround it. This is precisely what a memory system built around targeted retrieval accomplishes, and it’s the default behavior in Weaviate Engram rather than something bolted on afterward.
Consider a research assistant agent that has helped a user work through dozens of papers over several months. Instead of resending a running summary of every paper discussed so far, hoping the one relevant detail doesn’t get buried, the agent searches for exactly what’s relevant to the current question:
results = client.memories.search(
query="What did we conclude about batch size affecting convergence speed?",
user_id="researcher-208",
retrieval_config=HybridRetrieval(limit=5),
)
The five memories that come back are short, specific, and already filtered for relevance, which means there’s no long stretch of unrelated conversation for the important detail to get lost in the middle of. The position problem doesn’t need to be solved by clever prompt arrangement or by trusting a larger model to handle it better, because the underlying condition that causes the problem, a long, mostly irrelevant block of text with one useful fact hidden somewhere inside it, never gets created in the first place. Memory retrieval, done well, isn’t just about finding the right information. It’s about handing the model so little else along with it that where that information sits stops being able to matter.
Position isn’t the only way a context window can be a poor substitute for real memory, though. Even a request built entirely from short, targeted, well-ranked retrieval results still lives inside the same fundamental split between what the model holds temporarily during one call and what an agent needs to carry forward across many of them. Our next chapter, What is the difference between working memory and long-term memory?, looks directly at that split, and at why treating both kinds of memory as one and the same is where so many agent designs quietly go wrong.