How do you measure context quality?

Naming failure modes helps a team recognize what’s gone wrong once a system is already struggling. What’s still missing is a way to notice trouble before it becomes visible in obviously bad output, since several of the failures already covered in this Part produce confident, coherent-looking responses that hide exactly how broken the underlying context actually was. Measuring context quality directly, rather than judging a system purely by whether its final answers look reasonable, is what closes that gap.

Why Isn’t Judging Final Output Quality Enough on Its Own?

A model given poor context doesn’t produce visibly poor output, it produces fluent, confident output that simply happens to be wrong or ungrounded, exactly the hallucination behavior covered earlier in this Part. This means watching only the final response for problems misses failures that are actively occurring one layer upstream, in what got retrieved and assembled before generation ever happened. By the time a bad final answer is visible enough to notice, the underlying context failure that produced it may have been happening silently for a long time already.

This is why context quality needs its own dedicated measurement, separate from whatever evaluation gets applied to final responses. A system can score well on end-to-end output quality some of the time purely by luck, getting away with weak context because a particular question happened to be simple enough not to expose it, while still having a genuinely fragile retrieval layer that will eventually produce a visible failure on a harder question.

What Does It Mean to Measure Precision and Recall Specifically for Retrieved Context?

Precision asks what fraction of what got retrieved was actually relevant to the question being asked, and low precision means noisy, tangential material is crowding into context alongside whatever’s genuinely useful. Recall asks the opposite question, what fraction of everything that was actually needed to answer the question correctly made it into the retrieved set, and low recall means something essential got left behind, forcing the model to guess at the gap. A system can score well on one of these and poorly on the other, retrieving five results that are all narrowly on-topic but still missing the one specific fact the question actually needed, or retrieving broadly enough to capture that fact but burying it among a dozen barely-related results.

Neither metric alone tells the complete story, which is exactly why both need tracking together rather than picking whichever one happens to look better for a given system.

What Does Faithfulness Measure, and Why Is It the Metric That Actually Ties Everything Together?

Faithfulness measures whether the generated response is actually supported by the retrieved context, rather than drifting beyond it into unsupported claims the model added on its own. This is the metric that connects retrieval quality directly to grounding, since a system can retrieve genuinely good, relevant, complete context and still produce an unfaithful response if the model wasn’t sufficiently constrained to stick to what it was actually given. Faithfulness catches exactly the failure the grounding chapter earlier in this Part described: retrieval doing its job correctly while the generation step still embellishes beyond it.

Should These Measurements Be a One-Time Audit or an Ongoing Practice?

A one-time audit only captures quality at the specific moment it was measured, and every part of this knowledge base’s earlier chapters, from drift to stale content to reconciliation, described ways quality can degrade gradually and invisibly over time as a memory store or a knowledge base keeps changing. Treating context-quality metrics as something checked once during initial development, then never revisited, means a system that passed its original evaluation can silently degrade for months without anyone noticing until a visible failure finally forces the issue.

Establishing these metrics as a continuous, tracked baseline, checked before and after any change to the underlying pipeline, treats context quality with the same seriousness typically given to latency or cost, tracked numbers that get watched over time rather than assumed to stay fine once they looked fine once.

How Does Weaviate Engram’s Structure Support Measuring Retrieval Quality Directly?

Weaviate Engram’s search results come back with individual relevance scores per memory, giving a system exactly the raw material needed to measure precision and recall directly against a held-out set of representative queries, rather than having to infer retrieval quality indirectly from final output alone. Consider a specialty-coffee roastery’s cupping and quality-control assistant, where the retrieved context genuinely matters for consistent, defensible tasting notes across many batches:

from engram import EngramClient

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

client.memories.add(
    "This lot's washed processing produced noticeably brighter acidity than the natural-processed lot from the same farm last harvest, closer to a citrus profile than the stone-fruit character typical of this region.",
    properties={"lot_id": "guatemala-huehuetenango-24b"},
)

Measuring this retrieval’s quality means checking it against a representative set of real cupping questions, not just trusting that whatever comes back looks reasonable:

results = client.memories.search(
    query="How does this lot's acidity profile compare to prior harvests from the same farm?",
    properties={"lot_id": "guatemala-huehuetenango-24b"},
    retrieval_config=HybridRetrieval(limit=5),
)

Running this same query, and dozens of others representative of how cuppers actually phrase questions, against a held-out set with known correct answers lets the roastery measure precision, did the five results returned actually address acidity and prior-harvest comparison, and recall, was the specific comparative detail about processing method actually present among them, rather than assuming the system works well just because a handful of spot checks looked fine. Tracking these numbers over time, especially after any change to how lots get logged or described, catches a gradual quality decline long before it shows up as a cupper receiving a confidently wrong comparison they have no easy way to catch on their own.

This Part has built a complete discipline around getting the right information in front of a model at the right moment, from deciding what deserves inclusion through to measuring whether that discipline is actually working. Retrieval has come up constantly throughout, always assumed to run against a well-organized vector index without asking what actually makes that underlying search mechanism work. Our next chapter, What is a vector embedding?, opens the next Part by taking up exactly that foundation.