Short answer: Precision is how much of what you retrieved was useful; recall is how much of what you needed you found; relevance is whether helpful memories ranked first.
Memory search can look busy while still being wrong—five results with two that help and three that distract, or nothing useful while the needed fact sits deeper in the store. This chapter defines Precision@k and Recall@k for agent memory, explains why raising k often boosts recall while hurting precision, and shows how to score relevance on Engram search results. False positives show up as distracting context; false negatives as forgotten constraints. Practical tuning compares hybrid, vector, and BM25 on the same fixtures (hybrid is usually the default), and a costume-shop example computes Precision@3 and Recall@3 against labeled gold memories. Improving topic descriptions, deduping clutter, and aging gold labels when policies change keep the metrics honest.
Memory retrieval can look busy while still being wrong. The agent pulls five memories. Two help. Three distract. Or it pulls nothing useful while the needed fact sits one page deeper in the store. Precision, recall, and relevance are the language for diagnosing those failures. Precision asks how much of what you retrieved was useful. Recall asks how much of what you needed you actually found. Relevance asks whether ranking put the helpful memories first. This chapter defines those ideas for agent memory, shows how they trade off under different limits, and demonstrates how Weaviate Engram search configs let you measure and tune them on real fixtures.
What Do Precision and Recall Mean for Memory Search?
In classic information retrieval, a query has a set of relevant documents. Precision at k is the fraction of the top k results that are relevant. Recall at k is the fraction of all relevant documents that appear in those top k. Agent memory uses the same math with memories as documents.
Low recall means false negatives. The store holds the costume size note, but search never surfaces it. The agent invents a guess or asks the user again. Low precision means false positives. Search returns thrift-shop gossip beside the true measurement. The model may overweight noise and book the wrong rack.
Relevance is the graded middle. A memory can be partly related without being the answer. Ranking quality matters when several memories are weakly related and only one is decisive. High precision with terrible ordering still wastes the prompt’s best slots.
Why Does Raising k Improve Recall While Hurting Precision?
Once the definitions are clear, the limit knob becomes obvious. Engram’s retrieval_config takes a limit. Larger limits give recall more chances. Smaller limits protect precision and token budget. There is no free lunch. A limit of twenty may find a buried fact and also flood the prompt with near-misses.
Agents feel that tradeoff as behavior, not as a spreadsheet. With a tiny k, they sound forgetful. With a huge k, they sound confused. Good evaluation reports both Precision@k and Recall@k for the same fixture set. Optimizing only one metric invites the other failure mode.
Scope filters change the curve. Searching one shop rack property shrinks the candidate set. Precision often rises because unrelated shows never enter the pool. Recall can rise too if the gold memory lives in that scope. Wrong scopes create artificial false negatives that no ranking tweak will fix.
How Should You Score Relevance on Engram Search Results?
Metrics need labels. For each eval query, mark which memories are must-have, nice-to-have, or irrelevant. Must-haves drive recall. Nice-to-haves can inform graded relevance. Irrelevant hits count against precision. Keep labels tied to memory ids when possible so merges and updates do not silently invalidate the set.
Weaviate Engram returns a score on search hits. Treat that score as a ranking signal to study, not as automatic ground truth. Your human or rubric labels remain the authority. Compare hybrid, vector, and BM25 on the same queries. Hybrid is the recommended default for most Engram applications because it blends semantic matches with exact terms. Exact policy codes often need the keyword side. Soft preference language often needs the vector side.
Here is a costume-shop fixture that computes Precision@3 and Recall@3 against labeled gold memories:
import os
from engram import EngramClient, HybridRetrieval
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
user = "wardrobe-lead-sam"
group = "costume_shop"
rack = "costume-rack-19"
run = client.memories.add(
"Actor Mira needs a 42-regular frock coat from costume-rack-19. "
"Do not substitute the 40-regular even if it looks similar. "
"She has a wool allergy, so skip any wool blend linings.",
user_id=user,
group=group,
properties={"rack_id": rack},
)
client.runs.wait(run.run_id)
# After labeling, gold_ids are the memory ids that must appear for this query
query = "What coat size and fabric constraints apply for Mira on rack 19?"
hits = client.memories.search(
query=query,
user_id=user,
group=group,
properties={"rack_id": rack},
retrieval_config=HybridRetrieval(limit=3),
)
# Example: replace with ids recorded from your labeling pass
gold_ids = {"REPLACE_WITH_LABELED_MEMORY_ID"}
retrieved_ids = [m.id for m in hits]
relevant_hits = [mid for mid in retrieved_ids if mid in gold_ids]
precision_at_3 = len(relevant_hits) / max(len(retrieved_ids), 1)
recall_at_3 = len(relevant_hits) / max(len(gold_ids), 1)
print(
{
"precision@3": precision_at_3,
"recall@3": recall_at_3,
"scores": [(m.id, m.score, m.content) for m in hits],
}
)
If recall is zero while the fact exists under another rack id, fix scoping before you blame hybrid fusion. If precision is low because thrift notes keep ranking above the allergy constraint, tighten topics or rewrite queries to name the decision you need.
How Do False Positives and False Negatives Show Up in Agent Behavior?
Numbers become useful when mapped to symptoms. False negatives produce repeated questions, broken promises, and silent policy violations. Users say they already told the agent. Logs show empty or off-topic retrieval. That is a recall problem.
False positives produce confident mistakes. The agent cites a memory about a different production’s coat. Or it applies last season’s dye rule to this fitting. The retrieval set looked rich. Precision was the real failure. Utilization bugs are different again. Relevant memories were retrieved and the model still ignored them. Do not retune search until you separate those cases.
A simple diagnostic helps. For each failed turn, ask whether any retrieved memory was sufficient. If none were, label retrieval failure. If at least one was sufficient, label utilization failure. That split keeps ranking work from becoming prompt work by accident.
What Practical Tuning Moves Usually Improve the Metrics?
After diagnosis, tune with intent. Prefer hybrid retrieval for mixed natural-language and identifier queries. Use BM25 when the query is dominated by exact codes like rack ids. Use vector when the user paraphrases. Keep limits small in production prompts. Raise limits in offline recall probes when you need to learn whether the fact is findable at all.
Improve the store, not only the search. Better topic descriptions yield cleaner extracted memories. Cleaner memories raise both precision and recall because relevance becomes easier to judge. Deduped updates reduce near-duplicate clutter that steals top-k slots.
Revisit labels when policies change. Yesterday’s relevant memory can become irrelevant after a wardrobe rule update. Stale gold sets create fake regressions. Precision, recall, and relevance only stay honest when the labels age with the product.
Our next chapter, How do you benchmark long-term conversational memory?, zooms out from single-query metrics to multi-session conversational benchmarks that stress memory across weeks of dialogue.