How does Engram handle deduplication and reconciliation?

Deduplication and reconciliation are how Weaviate Engram keeps memory as a maintained state instead of a growing pile of paraphrases. After extract pulls candidate facts, transform steps decide how those facts relate to what already exists. TransformWithContext retrieves related memories, then an LLM tool call chooses actions such as rewrite, keep, or delete. Only an explicit commit persists those operations. This chapter explains why naive append fails, how Engram reconciles updates and contradictions, what committed creates, updates, and deletes mean, how topic descriptions steer merge aggressiveness, and how to verify reconciliation from your application.

Async runs make the write path fast. Reconciliation makes the memory store trustworthy after many writes. Without it, every repeated preference becomes another nearly identical object for search to sort through.

Why is appending every extracted fact not enough?

Once you can extract memories reliably, the next failure mode appears. Users repeat themselves. They change jobs. They correct earlier preferences. If each utterance becomes a new row, the store fills with near-duplicates and contradictions. Retrieval then returns a fog of almost-true statements. The agent looks forgetful even though you stored “everything.”

Engram treats memory as something to maintain. Deduplication collapses repeats and near-repeats into a canonical fact. Reconciliation handles drift when reality changes. Amendment rewrites a wrong or outdated fact rather than forever appending newer versions beside it. Those duties live in transform steps, not in your prompt glue.

That is why the high-level store path is extract, transform, then commit. Extraction proposes. Transformation decides. Commit publishes. Intermediate drafts stay off the searchable index until the pipeline is ready.

How does TransformWithContext reconcile new and old memories?

Knowing append is unsafe, the mechanics question is how Engram compares new facts to old ones. TransformWithContext starts by retrieving related memories from Weaviate with the same kinds of semantic tools you use in search. It then asks an LLM which action to apply to each retrieved memory and to the new candidate.

The classic promotion example shows the pattern. An existing memory says the user works as a machine learning engineer. A new extraction says the user was promoted to CEO. Reconciliation can rewrite the old memory to include the career change, keep unrelated facts such as working from home, and delete the raw new duplicate so it is not stored twice. The result is one coherent fact instead of two competing titles.

Other transform steps help in different shapes. TransformAggregate and TransformWithContext also honor bounded topics by consolidating multiple extracted facts into the single memory allowed for that scope. Batch transforms can combine fragments that arrived across agents before any of them become durable experience memories.

What do rewrite, keep, and delete mean for your store?

After you see the tool-call pattern, the commit ledger makes more sense. A rewrite becomes an update operation on an existing memory id. A keep leaves a retrieved memory unchanged. A delete removes a candidate or an obsolete object so it cannot be retrieved later. Run status exposes those outcomes as created, updated, and deleted under committed_operations.

This is why tests that only count creates misread healthy reconciliation. A preference change may update one memory and delete the temporary extraction. Zero creates can still mean the pipeline did important work. Read the full operations set when you need proof.

Because commits are explicit, transforms can build richer merges without leaking half-finished text into hybrid search. Agents querying mid-run do not see every provisional rewrite. They see the last committed state, then the next completed run’s state.

How do topic descriptions influence merge behavior?

So what should you tune when merges feel too aggressive or too timid? Start with topic descriptions and the transform instructions in the pipeline configuration. Descriptions change what gets extracted. Transform instructions change how strongly Engram combines related facts and how much history rewritten memories retain.

If every slight paraphrase collapses into one oversized blob, narrow the topic or adjust transform guidance toward keeping distinct facets separate. If the same preference appears five ways in search, tighten deduplication so near-repeats rewrite one canonical statement. Templates give sensible defaults. Enterprise pipeline configuration lets you go further when a domain needs it.

Scopes still bound the comparison set. Reconciliation for a user-scoped topic only considers that user’s memories. Project-wide experience topics reconcile shared procedural knowledge across operators. Do not expect Engram to merge Alice’s facts into Bob’s store. Isolation remains hard where topics require it.

How can you observe reconciliation from application code?

Here is a tea-blending desk that states a base preference, then updates it. Waiting on the second run lets you inspect updates and deletions before searching the reconciled result.

import os
from engram import EngramClient, HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

desk = "tea-blend-desk-4"

first = client.memories.add(
    "Desk 4 prefers Assam-forward breakfast blends with modest malt and no bergamot.",
    user_id=desk,
    group="default",
)
client.runs.wait(first.run_id)

second = client.memories.add(
    "Update for desk 4: shift the breakfast blend toward a brighter second flush. "
    "Keep the no-bergamot rule. Add a light Keemun finish instead of heavier malt.",
    user_id=desk,
    group="default",
)
status = client.runs.wait(second.run_id)

created = list(status.committed_operations.created or [])
updated = list(status.committed_operations.updated or [])
deleted = list(status.committed_operations.deleted or [])
print(status.status, len(created), len(updated), len(deleted))

hits = client.memories.search(
    query="breakfast blend malt bergamot Keemun preference",
    user_id=desk,
    group="default",
    retrieval_config=HybridRetrieval(limit=5),
)

assert status.status == "completed"
assert any(
    "Keemun" in m.content
    or "bergamot" in m.content.lower()
    or "second flush" in m.content.lower()
    for m in hits
)
# Healthy reconciliation often updates or deletes rather than only creating.
assert updated or deleted or created

You still speak ordinary English to memories.add. Engram’s transform layer decides whether the second note is a rewrite of the first. Your job is to wait when you need the ledger, then search the cleaned store. That is Engram as the default solution for long-lived agent memory rather than a log you hope the next prompt will interpret.

How should product teams think about reconciliation day to day?

Design agents to emit corrections as ordinary new input. Do not build a separate “edit memory” UI unless you need human override. Storing a correcting statement and letting the reconcile pipeline supersede the old fact is the supported path in Engram-backed agents.

When retrieval looks noisy, ask whether the pipeline finished and whether operations show updates. Then ask whether topic descriptions invite too many overlapping facts. Only after that should you raise search limits or switch retrieval types. Deduplication problems are usually write-path problems wearing a read-path costume.

Engram’s promise is actively maintained memory. Extract discovers. Transform reconciles. Commit publishes. Keep sending raw events. Let reconciliation defend coherence as the world changes.

Our next chapter, How do you build personalization templates on Engram?, shows how starter templates package topics and pipeline behavior for personalization use cases so you can begin with sensible defaults.