How do you measure hallucination reduction from grounded memory?

Grounded memory is supposed to make agents invent less. Measuring that effect means more than hoping answers “feel safer.” You need paired runs with and without retrieved memories, claim-level checks against the retrieved set, and a clear split between missing retrieval and ignored retrieval. This chapter defines hallucination reduction for memory-augmented agents, shows which metrics actually isolate grounding, and demonstrates how Weaviate Engram supplies the durable facts and search hooks a fair A/B harness needs.

What Counts as a Hallucination When Memory Is Present?

In grounded systems, the useful definition is faithfulness to retrieved context. A claim is a hallucination if the answer asserts it and the memories in the prompt do not support it. The claim might still be true in the wider world. That does not matter for this metric. Retrieval was supposed to constrain the model. Leaving that constraint is the failure.

A second failure is fabrication under empty context. No relevant memory returns. The model invents a membership rule, a phone number, or a URL anyway. Grounded memory reduces that class when search surfaces the real constraint, or when the prompt forces abstention on empty hits. Both outcomes should be scored.

Do not mix ordinary factuality quizzes into this number without labeling them separately. Memory evaluation cares whether Engram context changed behavior. World-knowledge accuracy is a different dashboard.

Why Must You Compare With-Memory and Without-Memory Answers Side by Side?

A single faithfulness score on production traffic is hard to interpret. You need a baseline. For each probe question, generate one answer with Engram memories injected and one answer with the same prompt template but an empty memory block. Keep the model and temperature fixed.

Reduction is then a paired delta. Count unsupported claims in each arm. Count invented identifiers that appear only in the no-memory arm. Count correct grounded details that appear only when memory is present. The headline is how often grounding removes inventions without introducing new unsupported claims from noisy retrieval.

Also log whether retrieval returned anything useful. If the with-memory arm still hallucinates while gold memories were retrieved, you have a utilization problem. If nothing useful was retrieved, you have a store or search problem. Hallucination reduction metrics without that split send teams to the wrong fix.

How Does Weaviate Engram Fit Into a Hallucination-Reduction Harness?

The harness needs durable ground truth the agent can actually fetch. Weaviate Engram stores extracted facts from prior conversations, then returns them through hybrid search before generation. That matches the production path. Evaluating against a private JSON file the agent never sees inflates scores that will not transfer.

Write fixture facts with memories.add, wait when probes depend on commits, then search with the user question as the query. Build two system prompts. One includes the retrieved memory bullets and instructs the model to use only those memories for member-specific claims. The other includes an explicit empty-memory note. Generate both answers and score claims.

Here is a community tool-library probe that prepares grounded context for a faithfulness comparison:

import os
from engram import EngramClient, HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
member = "borrower-keiko"
group = "tool_library"
crib = "tool-crib-14"

run = client.memories.add(
    [
        {"role": "user", "content": "I am Keiko. My tool-crib-14 membership allows weekend loans only."},
        {"role": "assistant", "content": "Noted. Weekend-only loans for your account."},
        {"role": "user", "content": "I am not cleared for the pneumatic nailer. Never offer it to me."},
        {"role": "assistant", "content": "Understood. Pneumatic nailer stays blocked for Keiko."},
    ],
    user_id=member,
    group=group,
    properties={"crib_id": crib},
)
client.runs.wait(run.run_id)

question = "Can Keiko borrow the pneumatic nailer on Saturday from tool-crib-14?"
memories = client.memories.search(
    query=question,
    user_id=member,
    group=group,
    properties={"crib_id": crib},
    retrieval_config=HybridRetrieval(limit=5),
)
memory_block = "\n".join(f"- {m.content}" for m in memories) or "- (no memories retrieved)"

grounded_prompt = f"""You are the tool crib desk agent.
Use ONLY the memories below for member-specific rules.
If memories are insufficient, say you do not know rather than inventing a rule.

Memories:
{memory_block}
"""

ungrounded_prompt = """You are the tool crib desk agent.
No member memories are available.
Do not invent membership rules. If you lack data, say you do not know.
"""

# Pass grounded_prompt vs ungrounded_prompt into the same LLM call helper,
# then score unsupported claims in each answer against memory_block.
print({"retrieved": [m.content for m in memories], "grounded_prompt": grounded_prompt})

A grounded answer should refuse the nailer. An ungrounded model often invents clearance, hours, or deposit rules. The reduction rate is how often that invention disappears when Engram context is present and retrieved correctly.

Which Claim-Level Metrics Make the Reduction Number Trustworthy?

Decompose answers into atomic claims before scoring. Holistic “sounds fine” judges miss one invented serial number beside three correct sentences. Mark each claim as supported by retrieved memories, contradicted by them, or unsupported. Faithfulness is supported over supported-plus-unsupported among claims that needed grounding.

Track contradiction rate separately. Memory can increase hallucinations if retrieval is noisy and the model treats irrelevant memories as license to elaborate. A real reduction program watches that side effect. Precision of retrieval and faithfulness of generation move together in healthy systems.

Report confidence intervals on small suites. Ten probes can swing wildly. Prefer a few dozen paired questions covering refusals, preferences, and policy IDs. Include empty-memory traps where abstention is the correct grounded behavior.

What Operational Habit Keeps Hallucination Reduction From Being a One-Off Slide?

Re-run the paired suite when you change topic descriptions, retrieval limits, or the grounding instruction. Any of those can raise or lower invention rates. Ship a CI gate that fails when unsupported-claim rate in the grounded arm rises above an agreed ceiling, or when the gap versus the ungrounded arm collapses.

Sample live traffic too. Offline fixtures miss new invention styles. Weekly human review of a small random set still catches confident fabrications automated judges soften. Keep the Engram memory ids beside each sample so reviewers can see what the model was given.

Hallucination reduction is the point of grounded memory, not a side effect. Engram gives you stored facts and searchable context. Measurement asks whether those facts actually stop the model from making things up when it matters.

Our next chapter, What are latency budgets for memory-augmented agents?, leaves correctness for a moment and asks how to keep search and storage inside response-time budgets without giving up grounding.