Short answer: It is the three-stage path from raw input to durable memory: pull facts, reconcile them, then save once.
Extraction understands raw content. Transform compares new facts to existing memory and settles coherent changes. Commit persists only after those decisions, so search never sees a half-updated state. The pipeline usually runs asynchronously so callers are not blocked on reasoning work. Engram structures memory creation as this extract, transform, commit sequence.
The previous Part covered how memory gets shaped once it exists, objects, schemas, relationships, and everything in between. This Part turns to a different question: how memory actually comes into being in the first place, the concrete sequence of steps that carries a piece of raw, unstructured data from its arrival all the way to durable, searchable storage. That sequence is a pipeline, and this chapter introduces its three core stages.
Why Does Turning Raw Data into Stored Memory Need to Happen in Distinct, Separate Stages Rather Than All at Once?
Each stage of this process does something genuinely different, extracting facts requires understanding raw content, integrating those facts with what’s already known requires comparing them against existing memory, and actually persisting the result requires committing to storage in a way that can’t easily be undone. Bundling all of this into a single, undifferentiated step would make each individual concern harder to reason about and harder to configure independently. Splitting the process into distinct stages lets each one be understood, tuned, and even swapped out on its own, without disturbing how the others work.
What Actually Happens During the Extraction Stage?
Extraction is the pipeline’s entry point, the stage that reads raw input, a conversation, a block of freeform text, or already-structured facts, and identifies the specific, durable pieces of information actually worth remembering. Different kinds of input shape naturally call for their own extraction approach, a multi-turn conversation needs to be read with an awareness of dialogue and role, while a plain block of text doesn’t carry that same structure and gets handled differently. Whatever the input’s shape, extraction’s job stays the same: identify what’s actually worth keeping and hand it forward as a set of candidate memories.
What Actually Happens During the Transform Stage, and Why Can’t Extraction Just Handle Everything on Its Own?
Extraction only knows about the raw input directly in front of it, it has no visibility into what’s already been stored from earlier conversations or earlier events. Transformation is where that missing context enters the picture, comparing freshly extracted candidates against existing memory to decide whether each one is genuinely new, a duplicate of something already known, or an update that should supersede an older fact. This is exactly where the deduplication, reconciliation, and identity-resolution concerns covered throughout the previous Part actually get applied in practice, not as abstract principles but as concrete decisions made about each specific new fact as it arrives.
Why Does Nothing Actually Get Saved Until the Final Commit Stage, Rather Than as Soon as a Decision Is Made?
Keeping the transform stage’s decisions separate from actually persisting them means a pipeline can work through several extracted facts, compare them against each other and against existing memory, and settle on a final, coherent set of changes before any of it becomes visible to a search. If committing happened immediately after each individual decision, a search running mid-pipeline could see a partially updated, inconsistent state, a new fact stored without yet accounting for whether it duplicates something else. Committing only once the transform stage’s full set of decisions is settled avoids exposing that kind of half-finished, potentially misleading intermediate state to anyone searching memory at just the wrong moment.
Why Does This Entire Process Run Asynchronously Rather Than Making a Caller Wait for It to Finish?
Extraction and transformation both typically involve real reasoning work, comparing new information against existing context and making judgment calls about how to integrate it, and that work takes measurably longer than a simple, immediate database write would. Forcing a caller to wait for the entire pipeline to finish before continuing would introduce real, unwanted latency into whatever application actually depends on quickly recording new information. Running the pipeline asynchronously lets a caller hand off raw data and move on immediately, trusting the pipeline to finish its work reliably in the background rather than blocking on it directly.
How Does Weaviate Engram Structure This Extract, Transform, and Commit Sequence in Practice?
Weaviate Engram processes every piece of incoming content through exactly this three-stage pipeline, returning a trackable run identifier immediately so a caller never has to wait for the full sequence to complete before continuing. Consider a home health equipment rental company’s customer-interaction assistant, helping coordinators track conversations with patients renting mobility and respiratory equipment:
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
run = client.memories.add(
[
{"role": "user", "content": "The oxygen concentrator delivered yesterday seems to be running louder than the previous unit we had."},
{"role": "assistant", "content": "Thanks for flagging that. I'll note it for our technician to check during the next scheduled visit."},
],
properties={"customer_id": "customer-rental-6634"},
)
print(run.run_id)
print(run.status)
Behind that single call, extraction identifies the durable, service-relevant fact buried in this exchange, the concentrator’s unusual noise, transformation checks whether this equipment already has a related open note that this new observation should merge into rather than duplicate, and commit persists whatever the transform stage decided, all without the calling application waiting on any of it to finish before moving on to its next task:
status = client.runs.wait(run.run_id)
print(status.committed_operations)
A coordinator can later search this customer’s equipment history and reliably find the noise complaint alongside anything else relevant, precisely because the pipeline’s transform stage already reconciled it against whatever else was on file rather than leaving two disconnected, overlapping notes for a technician to sort through manually. This three-stage structure, extract, transform, commit, is exactly what turns a raw, in-the-moment conversation into memory a coordinator can actually rely on later.
This chapter introduced the pipeline’s three stages at a high level. Each one deserves a closer look on its own, starting with extraction, the stage responsible for actually pulling durable facts out of the often messy, conversational raw material a system receives. Our next chapter, How does extraction pull facts from conversation?, takes up exactly that stage in detail.