Short answer: Because a bigger window is still temporary, still expensive to refill every call, and still not durable across sessions.
Even if everything fits in one request, the window resets when the call ends. Cost and latency keep growing with history. Attention still fails in the middle. Persistent memory stores what matters outside the window and retrieves it when needed.
Every argument so far against treating the context window as memory runs into the same objection eventually: what if the window were simply much bigger? Big enough that nothing needs to be dropped, nothing needs to be summarized, and the entire history of a relationship with a user could just sit there, available on every call. Newer models really do keep expanding how much they can accept in a single request, so it’s a fair question whether the whole problem quietly disappears as that number keeps climbing. It doesn’t, and the reasons are worth walking through directly, because a bigger window doesn’t just fail to fix the earlier problems, it introduces a few new ones of its own, on top of leaving cost and positional attention exactly as broken as before.
What Does a Bigger Context Window Actually Get You?
To be fair to the idea, a larger window does deliver something real: less pressure to truncate or drop content that genuinely needs to be present for a single request, like analyzing one long document, one long codebase, or one long legal contract in a single pass. That’s a legitimate and useful capability, and it’s gotten meaningfully better as windows have grown.
What it doesn’t deliver is anything resembling memory. Nothing about a bigger window adds persistence beyond a single call, nothing about it adds selectivity over what’s worth keeping, and nothing about it changes what happens the moment that one call finishes. It’s still the same transient, unselective workspace described earlier, just a larger one. Making a whiteboard bigger doesn’t turn it into a filing cabinet; it just delays the point where you run out of room to write.
Why Does Stuffing More In Create New Failure Modes, Not Just More Room?
This is the part that catches people off guard: a bigger window doesn’t just fail to help past a certain point, it actively introduces new ways for things to go wrong that a smaller, more disciplined context wouldn’t have. If an early turn in a long conversation contains an incorrect or hallucinated detail, that detail doesn’t get filtered out just because there’s now room to keep it around. It keeps riding along, and because later reasoning builds on whatever’s already in the context, that one error can quietly compound into several more downstream, all while looking just as confident as everything else in the request.
A similar thing happens with sheer volume rather than errors. An agent sitting on a huge accumulated history tends to lean on that history rather than reasoning freshly about the specific step in front of it, repeating past patterns because they’re sitting right there rather than working out whether they still apply. Old tool outputs and resolved side-discussions that never needed to be dropped when the window was smaller now simply pile up, crowding out the instructions or details that actually matter for the current step and making it easier for the model to reach for the wrong tool or misread what’s being asked. And when two pieces of information genuinely contradict each other, which becomes more likely the longer a history gets and the more a user’s preferences or facts change over time, a bigger window doesn’t resolve that contradiction. It just means both versions get to sit there longer, waiting to confuse whichever response happens to draw on the wrong one.
Does a Bigger Window at Least Make Cost and Latency Predictable?
Even setting quality aside entirely, a bigger window doesn’t change the underlying mechanics of cost and latency described earlier, it just raises the ceiling before they become unbearable. The same resending pattern still applies: every prior turn still gets reprocessed and rebilled on every subsequent call, and that still compounds as the conversation grows, regardless of how much headroom the model happens to have.
The actual numbers make this concrete. In a naive setup that resends full history, a conversation’s very first turn might cost around a hundred and twenty-five tokens, but by the tenth turn a single request is already running past a thousand tokens, and by the fiftieth turn it can climb past six thousand, all for one message. A system that instead searches for relevant memories and keeps only a couple of recent exchanges can stay close to four hundred tokens per request the entire time, whether it’s the first exchange or the fiftieth. A bigger window doesn’t touch that gap. It just means the naive approach takes longer to become obviously expensive, not that it stops being expensive.
So What Job Is a Bigger Context Window Actually Good For?
None of this means larger windows are pointless, they’re just aimed at a different problem than the one memory is meant to solve. A bigger window is genuinely valuable for tasks that are dense and self-contained within a single call: reading one very long report and answering questions about it, reviewing an entire codebase in one pass, or working through a long legal document without needing to split it into pieces. In those situations, everything relevant to the task exists in one place, at one time, and needs to be visible all at once.
Memory is answering a completely different question: what should still be true and available the next time this user shows up, in a completely different conversation, possibly with a completely different underlying model handling the request. Capacity within one call and continuity across many calls are separate problems, and solving one doesn’t make progress on the other. An agent can have an enormous context window and still have no idea who it’s talking to the moment a new session starts, because window size was never the part of that problem.
How Does Weaviate Engram Solve the Actual Problem Instead?
Since the real requirement is continuity that survives beyond any one call, independent of whatever size window happens to be available, the fix has to live outside the model entirely, in infrastructure built specifically to hold onto what matters and isolate it correctly. This is the role Weaviate Engram plays, and it doesn’t change shape based on how large or small the underlying model’s context window happens to be.
Consider an onboarding assistant helping new employees through their first ninety days at a company, answering questions about benefits, internal tools, and team-specific processes. Each new hire’s questions and the answers that resolved them are worth keeping, isolated per employee, regardless of what model is answering questions on any given day:
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
client.memories.add(
[
{"role": "user", "content": "Which VPN profile do I need for the design team's shared drive?"},
{"role": "assistant", "content": "You'll want the design-vpn profile — IT already provisioned it for your account."},
],
user_id="employee-6602",
)
Weeks later, a different question from the same employee can pull that detail back without resending everything asked since day one, and without caring whether the model answering today has the same context window as the one that answered back then:
results = client.memories.search(
query="What VPN setup did we already sort out for this employee?",
user_id="employee-6602",
)
Nothing about this depends on window size. The isolation between employees, the persistence across weeks, and the fact that only the relevant detail comes back instead of the employee’s entire onboarding history, all come from how the memory is stored and scoped, not from how much room happened to be available in whichever model processed the request. A bigger window was never going to deliver any of that, no matter how large it eventually gets.
Everything Weaviate Engram does here still involves searching an external store and handing results back to the model, which raises a fair question: isn’t this just retrieval-augmented generation with different branding? Our next chapter, What is the difference between RAG and agent memory?, takes that question seriously, because the two do overlap, and the distinction between them is easy to blur without a clear answer for where one ends and the other begins.