What are over-retrieval and under-retrieval?

Every chapter in this Part so far has focused on getting a specific piece of retrieval right, choosing the right retrieval type, setting the right threshold, scoping to the right user. This chapter steps back and looks at what actually goes wrong when retrieval fails despite all of that careful design, and it turns out failure comes in two genuinely opposite directions rather than one.

What Does It Actually Mean for a Search to Fail by Retrieving Too Much?

Over-retrieval happens when a search returns memories that don’t meaningfully help answer the question at hand, padding a result set with weakly related or entirely irrelevant material alongside whatever’s actually useful. This isn’t simply an efficiency problem, wasted space in a context window that could have held something better, it’s also a correctness problem, since a language model receiving that padded context has no reliable way to distinguish a genuinely load-bearing memory from noise that happened to come along for the ride. Noise doesn’t sit passively in the background, it actively competes with genuine signal for the model’s attention.

Why Does a Language Model Actually Struggle to Simply Ignore Irrelevant Retrieved Content on Its Own?

A model receiving a mix of genuinely relevant and irrelevant retrieved memories doesn’t reliably separate the two on its own, it tends to draw on whatever’s present in its context regardless of how weak that content’s actual connection to the query really is. This is exactly why enforcing a similarity threshold, discussed several chapters back, matters as much as it does, a search that returns a fixed number of results no matter what will happily hand a model several results with no real connection to the query at all, rather than confidently telling the caller nothing good enough was found.

What Does the Opposite Failure, Retrieving Too Little, Actually Look Like in Practice?

Under-retrieval happens when a search misses memories that genuinely mattered for answering the question, either because a threshold was set too strictly, because a query’s exact wording drifted too far from how the relevant memory was actually phrased, or because a question required following a chain of connections that a single, one-shot search was never equipped to make. A model facing this kind of gap doesn’t typically respond by admitting it lacks the information, it tends to fill the gap using its own general training instead, producing an answer that sounds just as confident as one grounded in real, retrieved memory.

Why Are Both of These Failures Genuinely Hard to Notice While a System Is Actually Running?

Neither failure announces itself. An over-retrieval failure doesn’t come with an error message, it comes with a subtly diluted or occasionally wrong-feeling answer that’s easy to mistake for an ordinary quality issue rather than a retrieval problem specifically. An under-retrieval failure is arguably worse in this respect, since a model papering over a genuine gap with fabricated, confident-sounding content produces output that’s often indistinguishable on the surface from output grounded in solid retrieval. Catching either failure reliably tends to require deliberately comparing what a search actually returned against what a human reviewer would agree was actually needed, rather than trusting a system’s own confident tone as a signal that everything went well.

Is There a Practical Way to Tell Which of These Two Failure Modes Is Actually the Bigger Risk for a Given System?

It genuinely depends on the specific use case and what a wrong answer actually costs in that context. A system where an incomplete answer is a minor inconvenience, easily corrected in a follow-up turn, can often tolerate some risk of under-retrieval in exchange for keeping searches fast and tightly scoped. A system where a wrong or fabricated answer carries real consequences, a compliance judgment, a medical suggestion, a financial calculation, generally deserves to lean the other way, accepting a little more risk of over-retrieval’s diluting noise in exchange for a much lower chance of confidently answering from something that was never actually there.

How Does Weaviate Engram Help a System Catch and Correct Both of These Retrieval Failure Modes?

Weaviate Engram’s search results carry the same similarity scores needed to detect an over-retrieval risk directly, and its search parameters expose the levers needed to correct an under-retrieval gap once one’s identified. Consider a pharmacy benefits assistant helping a member understand what their insurance plan actually covers, where returning weak, unrelated coverage details or silently missing the one detail that actually matters both carry real consequences for a member making a decision about their own medication:

from engram import EngramClient
from engram import HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

results = client.memories.search(
    query="Does my plan cover the brand-name version of this medication or only the generic?",
    user_id="member-pharmacy-benefits-3391",
    retrieval_config=HybridRetrieval(limit=5),
)

genuine_matches = [m for m in results if m.score >= 0.6]
if not genuine_matches:
    fallback_needed = True

Filtering the raw results down to only those clearing a real similarity threshold guards directly against over-retrieval, keeping weakly related coverage details from diluting the member’s actual answer, and explicitly checking whether anything survived that filter guards against under-retrieval, flagging the case where nothing genuinely relevant came back so the assistant can say so honestly rather than guessing. This is exactly the value deliberately watching for both failure modes delivers for a use case like pharmacy benefits, where a member deciding whether they can afford a specific medication deserves an answer grounded in their actual plan, not one diluted by irrelevant coverage details or quietly invented to fill a gap the search never actually closed.

Recognizing over-retrieval and under-retrieval as two distinct, opposite failures makes it possible to actually watch for and correct each one deliberately, rather than treating every disappointing answer as an undifferentiated generation problem. Everything covered in this Part so far has assumed retrieval happens through whatever code directly calls a search method, but a growing number of systems are built on agent frameworks with their own conventions for how a tool like retrieval gets exposed. Our next chapter, How should you design retrieval APIs for agent frameworks?, takes up exactly that question.