Short answer: Consistency means retrieved memories can be believed together after updates and corrections—not a pile of rival sentences from weeks of dialogue.
Long sessions do not only test recall of old facts; they test whether memory stays coherent when preferences flip, policies are corrected, or two notes disagree. Consistency here is whether memories an agent would retrieve can be believed together: after an update the current answer should reflect the newest durable fact, and after a correction the false version should stop winning retrieval. This chapter covers consistency failures to inject on purpose, how Weaviate Engram’s transform-and-commit pipeline plus run inspection give measurable artifacts, and how to score without collapsing conflict into one number. A community darkroom timeline example updates chemistry guidance then probes for a consistent present-tense answer—after waiting for commits so early search does not create false inconsistency reports. Cadence matters: re-run after topic description changes, because extraction tweaks can weaken update merging.
Long sessions do not only test whether memory can recall old facts. They test whether memory stays coherent when facts change. A preference flips. A policy is corrected. Two notes disagree. Consistency evaluation asks whether the store ends in a usable state, not a pile of rival sentences. This chapter defines the consistency failures that appear over weeks of dialogue, shows how to probe updates and contradictions with labeled timelines, and demonstrates how Weaviate Engram’s transform-and-commit pipeline plus run inspection give you artifacts for measuring that coherence.
What Does Memory Consistency Mean Across a Long Timeline?
Consistency here is not distributed-systems consensus jargon. It is whether the memories an agent would retrieve can be believed together. After an update, the current answer should reflect the newest durable fact. After a correction, the false version should not keep winning retrieval. After an irreducible conflict, the system should not pretend there was never a disagreement.
Append-only logs fail this test quietly. Every version remains searchable. The model sees both “use D-76 at 1:9” and “switch to HC-110 dilution B” and invents a compromise. Last-write-wins without reconciliation can also fail if the new write never merges with the old memory id and both remain near the top of hybrid search.
Good consistency is maintained state. Engram pipelines extract, transform with existing context, and commit creates, updates, and deletes. Evaluation must watch those outcomes over a scripted timeline, not only the final chat reply.
Which Consistency Failures Should Your Harness Inject on Purpose?
Once the definition is clear, design cases that force the failure modes. Preference updates change a stable choice. Factual corrections fix something that was wrong. Reversals undo a change and ask whether stale evidence was revoked from active use. Parallel notes introduce two true-in-context facts that look contradictory without a time or situation tag.
For each case, write the expected post-state in plain language before you run anything. Example: after session three, the member develops only in HC-110 dilution B for film, not D-76. Your probe query must fail if D-76 still ranks as current guidance without historical framing. That expected state is the gold label for consistency, separate from generic recall labels.
Also include no-change controls. Repeating the same preference should deduplicate rather than spawn near-copies that steal top-k slots. Consistency includes resisting inflation, not only resolving drama.
How Does Weaviate Engram Help You Observe Consistency Mechanically?
Labels still need machinery. Weaviate Engram processes new conversation batches asynchronously. Transform steps can rewrite related memories, keep unchanged ones, and drop duplicates. When a run completes, committed_operations shows creates, updates, and deletes. That record is gold for consistency tests because you can assert that an update path fired instead of only hoping the text looks right.
In-order processing per scope matters for long timelines. Add session batches in order for the same user and wait when later probes depend on earlier commits. Searching too early creates false inconsistency reports. Searching after wait lets you judge the real store.
Here is a community darkroom timeline that updates chemistry guidance and then probes for a consistent current answer:
import os
from engram import EngramClient, HybridRetrieval
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
member = "printer-ravi"
group = "darkroom_ops"
bay = "darkroom-bay-7"
week_1 = [
{"role": "user", "content": "For darkroom-bay-7 I develop Tri-X in D-76 at 1:9 for nine minutes."},
{"role": "assistant", "content": "Logged. D-76 1:9, nine minutes for your Tri-X at bay-7."},
]
week_4 = [
{"role": "user", "content": "Change that. I moved to HC-110 dilution B for seven minutes. Stop recommending D-76 for me."},
{"role": "assistant", "content": "Updated. HC-110 dilution B for seven minutes is your current Tri-X process."},
]
for session in (week_1, week_4):
run = client.memories.add(
session,
user_id=member,
group=group,
properties={"bay_id": bay},
)
status = client.runs.wait(run.run_id)
assert status.status == "completed"
hits = client.memories.search(
query="What developer and time should Ravi use for Tri-X at bay-7 right now?",
user_id=member,
group=group,
properties={"bay_id": bay},
retrieval_config=HybridRetrieval(limit=5),
)
blob = " ".join(m.content.lower() for m in hits)
current_ok = ("hc-110" in blob or "dilution b" in blob) and "seven" in blob
stale_dominant = ("d-76" in blob) and ("hc-110" not in blob and "dilution b" not in blob)
print(
{
"current_guidance_present": current_ok,
"stale_only_failure": stale_dominant,
"sample": [m.content for m in hits],
}
)
A consistent store surfaces HC-110 as current. A brittle store keeps D-76 alone at the top. Inspect committed_operations on the week-four run when you need to see whether Engram updated or created memories during reconciliation.
How Should You Score Consistency Without Collapsing Conflict Into One Number?
Do not hide every outcome inside a single accuracy percentage. Score update success separately from stale-retention failures. Score duplicate inflation on repeat statements. Score unresolved-conflict cases on whether the agent asks for clarification instead of inventing a false synthesis.
For retrieval probes, require the current value and optionally allow historical phrasing that clearly marks the old value as past. “Used to use D-76, now uses HC-110” can be consistent. Two bare present-tense rivals usually are not. Write that rubric into the harness so judges do not improvise.
Separate utilization again. If retrieval is consistent and the model still recommends D-76, that is a prompt bug. Consistency evaluation of memory stops at the store and the retrieval set unless you explicitly open an answer-scoring track.
What Cadence Keeps Consistency Evaluation Honest as the Product Ages?
Refresh timelines when real policies change. Add reversal cases regularly. Stale-conflict research keeps showing that leaving revoked evidence active can make memory worse than no memory. Your suite should notice that regression early.
Keep fixtures in a dedicated Engram project. Run them after topic description changes, because transform behavior is guided by those descriptions. A wording tweak that improves extraction can accidentally weaken update merging. Consistency tests catch that class of surprise.
Long sessions earn trust when yesterday’s correction still governs today’s retrieval. Engram maintains memories through transforms and commits. Your evaluation must ask, on a schedule, whether that maintenance still produces one believable present tense.
Our next chapter, How do you measure hallucination reduction from grounded memory?, turns from internal consistency to external truthfulness, and asks how often grounded memories actually stop the model from inventing facts.