Short answer: It finds near-duplicate memories by vector similarity, not exact wording, then merges or updates instead of storing repeats.
People restate the same facts in different phrasing across conversations. Exact text matching misses those near-duplicates. Comparing embeddings catches the same meaning restated. The useful response is usually merge and enrich, not keep both or drop new detail. Thresholds that are too loose risk merging related but different facts. Engram runs this in the memory pipeline so one current picture remains.
Atomic facts, as the previous chapter described, split conversation into small, focused memories. That same conversational pattern that produces good atomic facts also produces a very ordinary side effect: people repeat themselves. A preference mentioned once in January often gets mentioned again in March, phrased slightly differently but describing the exact same underlying fact. Left unmanaged, this repetition quietly fills a memory store with near-duplicate entries that all say roughly the same thing.
Why Does This Kind of Repetition Happen So Naturally in Real Conversation?
People don’t track what they’ve already told a system, and there’s no reason they should. A user who mentioned a scheduling preference weeks ago will mention it again the next time it becomes relevant, phrased however feels natural in that specific moment, with no awareness that an identical fact might already be sitting in storage from an earlier conversation. Extraction, working exactly as intended, will happily turn each of these separate mentions into its own separate memory unless something downstream actively recognizes that they’re saying the same thing.
Why Doesn’t Comparing the Exact Wording of Two Memories Reliably Catch This Kind of Duplication?
Two mentions of the same underlying fact are rarely phrased identically. One might state a preference directly, another might mention it in passing while explaining something else entirely, and a simple text comparison looking for identical or near-identical wording would miss the connection between them almost every time. This is exactly the gap semantic representation is built to close: two differently worded statements describing the same underlying fact land close together in vector space, even when their surface wording shares almost nothing in common, because their embeddings capture what they mean rather than the specific words used to say it.
How Does Comparing Vector Representations Actually Catch a Near-Duplicate That Wording Alone Would Miss?
When a new memory is extracted, comparing its embedding against the embeddings of existing, related memories reveals whether something very similar already exists, based on genuine semantic closeness rather than any requirement that the wording match. A newly extracted memory landing extremely close to an existing one, closer than genuinely distinct facts would ever land, is a strong signal that the two are actually describing the same underlying reality, phrased two different ways rather than genuinely representing two separate pieces of information worth keeping both of.
What Should Actually Happen Once Two Memories Are Recognized as Likely Duplicates?
Simply discarding the newer mention risks losing something the older memory didn’t capture, a helpful added detail, a slightly different phrasing that clarifies an ambiguous earlier version, or evidence that the fact has been reaffirmed and is still current. The more useful response usually merges the new information into the existing memory, updating or enriching it rather than either duplicating it outright or discarding new detail that might genuinely improve on what was already stored. This merging decision itself often benefits from an actual judgment call, deciding whether the new mention adds real value or is simply restating something already fully captured.
Does This Kind of Deduplication Risk Merging Together Things That Are Actually Genuinely Different?
It can, if the similarity threshold used to flag a likely duplicate is set too loosely. Two memories that are related but not actually identical, a preference for one thing and a related but distinct preference for something adjacent, can sit close enough in vector space to look like duplicates without genuinely being the same fact. Getting this threshold right is a real calibration exercise, set too aggressively it merges things that should have stayed separate, set too conservatively it lets genuine duplicates pile up unchecked, and the right balance depends on how tightly related content in a specific domain actually tends to cluster in that domain’s own embedding space.
How Does Weaviate Engram Handle This Deduplication Step as Part of Its Broader Memory Pipeline?
Weaviate Engram’s transform step queries for related existing memories using the same semantic search available through its normal retrieval API, then uses this comparison to decide whether a newly extracted memory should be kept separate, merged into something already stored, or discarded as a pure restatement. Consider a specialty coffee roastery’s supplier-relationship assistant, tracking notes from recurring calls with green coffee importers across many separate conversations over the course of a buying season:
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
client.memories.add(
"This importer mentioned their Yirgacheffe lot from the current harvest is running slightly later than usual due to unexpected rain during drying.",
properties={"supplier_id": "supplier-highland-imports"},
)
Weeks later, a follow-up call touches on the same delay again, phrased differently and adding a bit more detail:
client.memories.add(
"Follow-up call confirmed the Yirgacheffe delay is now about two weeks, tied to the drying issues from the earlier rain, with an updated arrival estimate.",
properties={"supplier_id": "supplier-highland-imports"},
)
Because Engram’s transform step compares this new mention’s embedding against existing memories for this same supplier before committing anything permanently, it recognizes these two entries as describing the same underlying delay rather than two unrelated facts, merging the updated timeline and estimate into the original memory rather than leaving two overlapping, increasingly redundant entries sitting side by side. A buyer searching this supplier’s history later gets one clear, current picture of the delay, its cause, and its latest estimate, rather than having to piece that story back together from several scattered, partially overlapping notes accumulated across the buying season.
Deduplication at the representation level keeps a memory store from quietly filling up with redundant restatements of the same underlying fact. Once memory stays this clean, a related and equally practical need emerges: giving each memory the kind of structured, filterable metadata that lets a caller narrow a search down precisely, beyond what semantic similarity alone can offer. Our next chapter, What is property-based metadata for memory filtering?, takes up exactly that need.