How do you decide what belongs in the context window?

Context engineering as a discipline and memory engineering as its own custodial specialty are now both clearly defined. What’s still missing is the practical, moment-to-moment mechanics: given a specific question a model is about to answer, and a memory store that might hold dozens or hundreds of relevant-sounding facts, how does a system actually decide which handful of them earn an actual spot in that call’s context window. This is where the abstract discipline becomes a concrete engineering problem with real, testable answers.

Why Is “Just Include Everything Relevant” Not a Workable Answer?

“Relevant” is a matter of degree, not a binary label. A semantic search against a memory store returns results ranked by similarity, and there’s rarely a clean cutoff where everything above some score is genuinely useful and everything below it is worthless. Including too much of that ranked list, reasoning that more relevant material can only help, runs directly into the failure modes already touched on earlier in this Part: contradictory or barely-related entries crowding the window, diluting the specific signal the model actually needs for this one question.

The practical answer isn’t “include everything relevant,” it’s “include the smallest set that actually changes what a good answer looks like.” That’s a genuinely different and more demanding standard, one that requires an actual decision rather than a blanket policy of maximal inclusion.

What Criteria Actually Distinguish Content Worth Including From Content That Isn’t?

Similarity to the current query is the obvious first filter, and it’s usually the primary signal a retrieval step relies on. But similarity alone misses two things that matter just as much: recency, since a fact that’s since been superseded shouldn’t outrank a more current one just because it happens to score similarly, and specificity, since a narrowly relevant fact directly answering the question at hand is worth more than several loosely related facts that each touch the topic without actually resolving it.

A useful mental test is to ask, for each candidate piece of content, whether removing it would actually change the quality of the answer the model produces. If a fact’s presence or absence makes no real difference, it’s taking up space in the context window without earning that space, regardless of how technically relevant it might have scored.

Does the Answer to This Question Change Depending on What Kind of Task Is Being Performed?

It does, considerably. A quick factual lookup usually needs very little context, often just one or two directly relevant facts retrieved with a tight limit. A complex, multi-step reasoning task, by contrast, might genuinely benefit from a broader set of context spanning several related facts, since the reasoning itself depends on connecting pieces that wouldn’t individually seem essential in isolation. Applying the same fixed retrieval limit to both kinds of tasks means either starving the complex task of context it actually needs, or flooding the simple task with more than it can productively use.

This is why retrieval limits and thresholds are rarely set once and left alone. They’re tuned against the specific kind of task a given call is serving, adjusted as real usage reveals where too little or too much context is actually hurting response quality.

What Happens When Nothing Retrieved Actually Clears the Bar for Inclusion?

This is a legitimate and common outcome, not a failure state to work around. If nothing in memory is genuinely relevant to the current question, the correct response is to proceed with an empty or minimal context rather than forcing in loosely related material just to avoid returning nothing. Including tangential content purely to fill space produces exactly the kind of distracting clutter this whole discipline is trying to prevent, and a model reasoning from a genuinely empty relevant-context set, honestly signaling that it doesn’t have specific prior information, is a far better outcome than one confidently working from material that doesn’t actually apply.

How Does Weaviate Engram Support Making This Inclusion Decision Deliberately?

Weaviate Engram’s search API exposes exactly the controls this decision requires: a choice of retrieval type suited to the kind of matching needed, a limit controlling how much comes back, and topic filtering to narrow the search to only the categories of memory actually relevant to the current task. Consider an independent bookstore’s personal-shopper assistant recommending books to regular customers, where a customer’s accumulated reading history could easily overwhelm a single recommendation request if included wholesale:

from engram import EngramClient

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

client.memories.add(
    "Loved the slow-burn mystery pacing in the last two books recommended; specifically disliked one recent thriller for being too gimmicky with its twist.",
    user_id="customer-4187",
)

When the customer comes in asking for a new mystery recommendation, a narrow, tightly limited search is exactly the right amount of context for this specific question, rather than retrieving everything ever known about the customer’s reading history across every genre they’ve ever mentioned:

relevant_preferences = client.memories.search(
    query="What has this customer liked or disliked about mystery novels specifically?",
    user_id="customer-4187",
    topics=["ReadingPreferences"],
    retrieval_config=HybridRetrieval(limit=5),
)

Limiting to five results, and filtering specifically to the reading-preferences topic rather than searching the customer’s entire memory store, are both deliberate inclusion decisions. If the customer had instead asked something requiring broader context, say, a request for a reading list spanning several genres for an upcoming vacation, that same decision would reasonably shift toward a higher limit and a broader topic scope, because the task itself calls for more connected context rather than one narrow answer. The right amount of context isn’t a fixed number, it’s a decision made fresh for each kind of request, based on what that specific request actually needs to be answered well.

Deciding what deserves inclusion is a qualitative judgment about relevance and usefulness. There’s a related, more quantitative constraint sitting right alongside it: even genuinely relevant content still has to fit within a hard token limit, and that budget has to be managed deliberately rather than assumed to always have room. Our next chapter, What is a context budget?, turns to exactly that constraint.