What is a context budget?

Deciding which content deserves inclusion answers a qualitative question about relevance and usefulness. Sitting right alongside that judgment is a harder, more unforgiving constraint: even content that clearly deserves inclusion still has to fit within a hard numeric limit, measured in tokens, that no amount of good judgment about relevance can stretch. Treating that limit as a real, finite budget rather than an afterthought is what this chapter is about.

What Actually Counts Against the Context Budget?

Every single component that reaches the model competes for the same finite space: the system prompt, the live conversation history, any retrieved memory, tool definitions describing what the agent can call, and the results those tool calls return. None of these categories gets some separate, protected allowance. A system prompt that’s grown bloated with instructions competes directly with retrieved memory for the exact same limited space, and a long tool result can just as easily crowd out conversation history as an overly long block of retrieved facts can.

This shared-budget framing matters because it’s tempting to think about each component in isolation, optimizing retrieval separately from prompt length separately from tool output size. In practice, every one of those optimizations draws from the same pool, and a win in one area only matters if it actually leaves more room for something else that needed the space.

Why Does Naively Growing Conversation History Consume the Budget So Quickly?

The most common way a context budget gets exhausted without anyone intending it is simply appending every message of a conversation to a growing list and resending the whole thing on every single call. Token usage under this approach grows roughly linearly with conversation length, since turn fifty resends everything from turns one through forty-nine on top of whatever’s new. A fifty-turn conversation handled this way can easily balloon past ten thousand tokens of input on every single call, almost none of which is new information relevant to what’s actually being asked right now.

Replacing that full history with a targeted memory search instead keeps the token cost roughly flat regardless of how long the underlying conversation has run, since a handful of retrieved facts plus a short window of recent messages costs about the same whether the conversation is five turns old or five hundred. The savings compound directly with conversation length: negligible at first, sizable by turn ten, and dramatic by turn fifty, since the token cost stops scaling with history at all.

Does a Larger Context Window Just Solve This Problem by Raising the Ceiling?

It raises the ceiling, but doesn’t change the fact that every token placed in the window still competes for the model’s attention and still costs something on every call. The failure modes already covered elsewhere in this Part, distraction, confusion, and clash, don’t require actually hitting the window’s hard limit to start degrading a response, they show up well before that ceiling, as soon as the window fills with material that isn’t earning its place. A bigger ceiling delays the moment content literally gets truncated, but it does nothing to prevent the quieter, earlier degradation that comes from treating the budget as effectively unlimited.

This is why budget discipline matters even for models with genuinely enormous context windows. The temptation to stop being careful once the ceiling feels distant is exactly the temptation that produces bloated, unfocused context long before that ceiling is ever actually reached.

How Should a Team Actually Allocate a Limited Budget Across Competing Components?

A workable approach treats the budget as something to explicitly divide, not something to fill opportunistically until it runs out. A fixed, small allowance for system instructions, since those rarely need to grow with the conversation, a modest, capped allowance for recent conversation history covering only the last few exchanges, and the remaining space reserved specifically for retrieved memory and tool results, sized according to what the current task actually needs. Deciding these allocations ahead of time, rather than letting whichever component happens to be verbose that day consume whatever space is left, keeps the budget from being silently dominated by whichever part of the system is least disciplined about its own size.

How Does Weaviate Engram Help Keep Memory’s Share of the Budget Under Control?

Weaviate Engram’s default behavior of extracting compact, atomic facts rather than storing raw conversation transcripts is itself a budget-conscious design choice, since a handful of short, precise facts costs far less than an equivalent stretch of unprocessed dialogue while often conveying more of what actually matters. Consider a marine cargo-shipping logistics dispatcher assistant coordinating vessel schedules across dozens of ongoing shipments, where an unmanaged approach could easily let context balloon out of control over a long operational day:

from engram import EngramClient

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

client.memories.add(
    "Vessel Meridian Star delayed departure by six hours due to port congestion at Rotterdam; downstream cargo transfer at Singapore now needs rescheduling.",
    properties={"vessel_id": "meridian-star-4471"},
)

Rather than replaying an entire day’s worth of dispatch chatter into every new query about this vessel, a tightly scoped, limited search keeps the memory portion of the budget small and precise:

relevant_updates = client.memories.search(
    query="What scheduling issues has this vessel had today?",
    properties={"vessel_id": "meridian-star-4471"},
    retrieval_config=HybridRetrieval(limit=5),
)

This search returns a handful of compact, atomic facts rather than a sprawling transcript of every message exchanged about the vessel that day, keeping memory’s share of the token budget small and predictable no matter how much operational chatter has accumulated behind the scenes. Combined with a capped recent-message window and a lean system prompt, the dispatcher’s context stays focused on exactly what’s needed to make the next scheduling decision, rather than growing unmanageably as the shipping day goes on. Treating the budget as something explicitly divided, rather than filled until something breaks, is what keeps a system usable even as the underlying operation it’s tracking keeps generating more history behind the scenes.

Managing the token budget is about how much content fits. A related but separate concern is how that content actually gets built and combined into a finished context for a given call, since simply knowing what to include and how much room it has doesn’t yet specify the mechanics of assembling it correctly. Our next chapter, What is a context assembly pipeline?, turns to exactly that process.