Short answer: Each store call returns a run id immediately; a DAG of steps extracts facts, transforms them against existing memories, optionally buffers, then commits creates, updates, or deletes.
Weaviate Engram turns raw input into durable memories through an asynchronous pipeline. Different content types enter through different extract steps; shared transform and commit stages often follow. Commit is the only persistence boundary—without that middle layer every add would either block the request path or dump unreconciled text into the index. Buffers accumulate work until count- or time-based triggers fire; while waiting, run status is in_buffer. A daily rollup or continual-learning flow can extract, transform, and commit immediately, then buffer and synthesize again later. Topics and scopes decide what and where; the pipeline decides how. Missing facts often point at topic descriptions; duplicates at transform; empty search right after add often means you did not wait for commit.
Weaviate Engram turns raw input into durable memories through an asynchronous pipeline. Each store call returns a run id immediately, then a directed graph of steps extracts facts, transforms them against what already exists, optionally buffers work, and commits create, update, or delete operations to storage. Different content types enter through different extract steps. Shared transform and commit stages often follow. This chapter walks that architecture, explains why commit is the only persistence boundary, how buffers pause a run, what run statuses mean, and how the Engram client lets you wait on a run when you need confirmation.
Topics and scopes decide what and where. The pipeline decides how. Without that middle layer, every add would either block your request path or dump unreconciled text into the index.
Why is the pipeline a DAG instead of a single write?
Once you can add memories with one API call, it is natural to wonder what happens after the response returns. Engram does not write final objects in that hot path. It starts a pipeline run. Pipelines are directed acyclic graphs. Each content type has its own extract entrypoint. Those entrypoints can converge on shared transform and commit steps, or take different downstream routes when a group needs that flexibility.
That shape matches real memory work. Extraction is not the same problem as reconciliation. Reconciliation is not the same problem as persistence. Separating them lets Engram use an LLM where language understanding helps, query existing memories where merge decisions need context, and only then touch Weaviate storage. Intermediate drafts stay off the searchable index until a commit step says they are ready.
Configurable pipelines are available on enterprise plans. Templates give you working graphs for common cases without hand-authoring steps on day one. The mental model stays the same either way. Raw data enters. Memories leave only after commit.
What happens in extract and transform?
Knowing the graph exists, the next question is what each major stage does. Extract is the entrypoint. ExtractFromString, ExtractFromConversation, and ExtractFromPreExtracted handle the three input types. String and conversation paths use the LLM with your topic descriptions as magnets. Pre-extracted items already carry a topic and skip LLM extraction, then still flow through transform and commit.
Transform refines the extracted batch. Steps such as TransformWithContext, TransformOperations, TransformConcatenate, and TransformAggregate deduplicate, merge, consolidate, and resolve conflicts. TransformWithContext can retrieve related memories from storage with the same kinds of semantic tools you use in search, then decide rewrite, keep, or delete actions. Bounded topics are honored here. Multiple extracted facts for one scope consolidate into the single memory that scope allows.
Those decisions are still provisional until commit. Pipelines can build richer memories across steps without leaking half-merged content into retrieval. That is why a promotion that updates an old job title can rewrite one memory and drop a duplicate instead of leaving two contradictory facts searchable.
Where do buffers fit between transform and commit?
Sometimes you need more than one add before the final memory is worth storing. Buffer steps pause the pipeline and accumulate memories or raw inputs until a trigger fires. Triggers can be count-based or time-based, such as time since the first item or time since the last item. Buffers can sit anywhere in the graph, not only at the start. Inputs from different content types that share a buffer are aggregated together.
A daily rollup pipeline might extract, transform, and commit immediately, then buffer those committed memories by scope, then transform and commit again into a “daily activity” memory when the buffer flushes. Continual-learning flows can hold partial task, action, and feedback memories until enough pieces exist to synthesize one experience memory. While a run waits on a buffer, its status is in_buffer.
Buffers are why Engram can learn from information that never appeared in a single context window. You still call add in small, low-latency batches. The pipeline decides when the batch is complete enough to finish.
What does a run represent for your application?
Each client.memories.add creates a run. A run is the trackable unit of pipeline execution. Status values are running, in_buffer, completed, and failed. When a run completes, committed_operations lists which memories were created, updated, or deleted, each with a memory id and timestamp.
Most production chat paths can fire and forget. Recent turns are already in the prompt. Eventual consistency is enough for the next session’s search. Use client.runs.wait or runs.get when tests, admin tools, or a tight read-after-write path need proof that a specific run finished. Failures surface an error string describing what went wrong.
Here is a cooper’s shop assistant that records stave notes, waits for the pipeline, and inspects committed operations before searching.
import os
from engram import EngramClient, HybridRetrieval
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
shop = "cooper-shop-east"
run = client.memories.add(
"Shop east toasted bourbon barrels for lot B-19 at a medium-plus char. "
"Keep bilge hoops snug after the second wetting. "
"Customer wants lower tannin bite than lot B-11.",
user_id=shop,
group="default",
)
print(run.run_id, run.status)
status = client.runs.wait(run.run_id)
print(status.status)
created = [op.memory_id for op in status.committed_operations.created]
updated = [op.memory_id for op in status.committed_operations.updated]
hits = client.memories.search(
query="char level and tannin target versus B-11 for lot B-19",
user_id=shop,
group="default",
retrieval_config=HybridRetrieval(limit=5),
)
assert status.status == "completed"
assert created or updated or hits
assert any(
"B-19" in m.content or "tannin" in m.content.lower() or "char" in m.content.lower()
for m in hits
)
The add returns before extract finishes. Waiting is optional and explicit. Search after completion sees the reconciled memories the commit step persisted, not the raw workshop paragraph alone.
How should you think about extract, transform, and commit as a product loop?
Architecturally, treat the three named stages as a contract with your agents. Extract pulls topic-matching facts from noisy events. Transform reconciles those facts with the living memory state. Commit publishes only the final operations. Buffers are the optional patience layer when facts arrive out of order or across agents.
That loop is why Engram can stay low latency on the request path while still doing serious memory hygiene offline. It is also why groups bundle a pipeline with topics. Changing how you process is a pipeline concern. Changing what you care about is a topic concern. Changing who can see it is a scope concern. Keep those knobs separate in your design notes.
When something looks wrong in retrieval, ask which stage failed the contract. Missing facts often point at topic descriptions or extract input shape. Duplicates and contradictions point at transform configuration. Empty search right after add often means you did not wait and the commit has not landed yet. Engram’s run model makes that last case observable instead of mysterious.
Our next chapter, How does durable workflow execution work behind Engram pipelines?, goes under the async surface. You will see how durable workflow execution keeps pipeline runs reliable and ordered even when workers restart.