What happens during the transform stage?

The previous chapter looked at extraction, the pipeline’s entry point, identifying which facts inside raw input are actually worth remembering. Extraction alone can’t finish the job, though, since it only ever sees the input directly in front of it, with no awareness of anything a system might already know. Transformation is the stage where that missing context finally enters the picture, deciding how each newly extracted fact should actually fit alongside everything already stored.

Why Can’t Extraction Simply Decide on Its Own Whether a New Fact Is Genuinely New?

Extraction reads whatever raw input it’s handed and identifies durable facts within that input, but it has no visibility into a system’s existing memory store, no way of knowing whether a fact it just identified already exists somewhere, phrased differently, from an earlier conversation. Without that missing context, extraction alone would happily produce a fresh, standalone memory every single time a topic came up again, regardless of how many times it had already been recorded before. Transformation exists specifically to close this gap, bringing existing memory into the picture before anything actually gets committed.

What Does the Transform Stage Actually Do with a Newly Extracted Fact Once It Arrives?

The transform stage retrieves memories already stored that seem related to the newly extracted fact, using the same kind of semantic search covered throughout this knowledge base’s earlier discussion of retrieval. With those related memories in hand, an underlying model compares the new fact against each one and decides, for every memory involved, exactly what should happen to it: stay exactly as it is, get rewritten to reflect updated information, or get removed because it’s now redundant or fully superseded. This comparison is where deduplication, reconciliation, and the other memory-maintenance concerns covered earlier in this knowledge base actually get applied to specific, concrete facts rather than existing only as abstract principles.

How Does This Comparison Actually Produce a Concrete Decision for Each Memory Involved?

The underlying model examines the new fact alongside each related existing memory and issues an explicit decision for each one: an existing memory that remains accurate as-is gets kept unchanged, one that needs updating gets rewritten to incorporate what’s new while preserving relevant history from before, and a newly extracted fact that turns out to be pure duplication of something already fully captured gets dropped entirely rather than stored redundantly alongside the memory it duplicates. This structured, explicit decision-making is exactly what keeps a growing memory store from silently accumulating conflicting or redundant entries every time a related topic comes up again.

Why Does It Matter That These Decisions Get Made Explicitly, Rather Than Simply Trusting the Newest Version to Automatically Win?

A naive approach might simply let the most recently extracted fact always overwrite anything related that came before it, but this loses exactly the nuance the transform stage is actually built to preserve, sometimes an older memory should stay untouched because it’s about a genuinely different aspect of the same broader subject, and sometimes a rewrite needs to weave old and new information together rather than simply replacing one with the other. Making each decision explicit, keep, rewrite, or delete, lets a system apply exactly the right handling to each specific memory involved, rather than defaulting to a blunt rule that would inevitably get some of these cases wrong.

Can the Transform Stage Do More Than Just Compare a Single New Fact Against What’s Already Stored?

Yes, transformation can also operate across an entire batch of newly extracted facts at once, without necessarily pulling in anything from existing memory at all, useful for consolidating several related pieces of information gathered from different parts of an interaction into one coherent memory before that memory ever gets compared against what’s already on file. This flexibility means transformation isn’t limited to one-at-a-time comparisons against history, it can also handle the more structural task of combining fragmented, freshly extracted information into a single, well-formed fact first.

How Does Weaviate Engram’s Transform Stage Apply This Kind of Integration in Practice?

Weaviate Engram’s transform step retrieves related existing memories and uses a structured LLM decision to determine exactly how each new fact should be integrated, keeping, rewriting, or dropping memories as appropriate. Consider a boutique hotel’s guest-preference concierge system, helping staff remember returning guests’ preferences across multiple separate stays:

from engram import EngramClient

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

client.memories.add(
    "Guest mentioned during check-in that they're now avoiding feather pillows entirely due to a recently developed allergy, a change from their previous preference for extra-plush bedding.",
    properties={"guest_id": "guest-loyalty-8827"},
)

Before this new fact ever gets committed, the transform stage retrieves this guest’s existing preference memory from an earlier stay and compares the two:

results = client.memories.search(
    query="What bedding and pillow preferences does this guest have?",
    properties={"guest_id": "guest-loyalty-8827"},
)

Because the transform stage recognizes that this new allergy detail genuinely updates, rather than merely duplicates, the guest’s earlier bedding preference, it rewrites the existing memory to reflect the current, accurate preference while dropping the newly extracted fact’s own separate, now-redundant version. A concierge preparing this guest’s next stay retrieves one clear, current memory reflecting the actual preference, rather than two overlapping, partially contradictory notes that leave staff guessing which one still applies. This is exactly the value the transform stage delivers: integration, not just accumulation, of everything a system comes to know over time.

Transformation is the pipeline’s stage for reconciling new information against everything already known, but one specific piece of that work, recognizing when two memories are simply restating the same underlying fact, deserves its own closer look. Our next chapter, How does deduplication work during transform?, takes up exactly that mechanism.