Short answer: Engram builds pipelines on Temporal so accepted work survives worker restarts, and runs for the same scope stay queued in submission order.
Durable workflow execution makes asynchronous pipelines trustworthy in production. memories.add returns a run id quickly; extract, transform, and commit continue in the background without inventing your own queues. In-order processing per scope protects reconciliation so preference updates stay serialized for that person. On the hot path, call add, keep the conversation moving, and let search pick up committed memories later—waiting on every run usually adds latency you do not need. Use runs.wait or runs.get for tests, migrations, or admin proof. AsyncEngramClient keeps request workers non-blocking while Engram workflows handle durability separately. When debugging “the agent forgot,” check run status and committed operations before rewriting topic descriptions—durability makes lag observable and silent loss rare.
Durable workflow execution is what makes Weaviate Engram’s asynchronous pipelines trustworthy in production. When you call memories.add, the API returns a run id quickly. The heavy extract, transform, and commit work continues in the background. Engram builds those pipelines on Temporal workflows so accepted work is not lost if a worker restarts. Runs for the same scope are queued in the order you submitted them. This chapter explains why durability matters for memory, how in-order processing protects reconciliation, what fire-and-forget means for your app, how run status exposes progress, and how Engram keeps the hot path free of memory I/O stalls.
Pipeline steps describe what should happen. Durable execution describes what still happens after crashes, retries, and bursts of writes. Memory systems without that layer force you to invent your own queues and recovery logic.
Why does memory need durable execution at all?
After you understand extract, transform, and commit, a practical fear appears. What if the process dies halfway through a merge? A naive background job can drop work or double-apply side effects. Engram’s answer is durable workflows. Once data has been successfully accepted, the pipeline is expected to finish, and the resulting Weaviate object changes are expected to complete.
That guarantee is why fire-and-forget is safe as a product pattern. Your chat handler does not need to keep a thread open while an LLM extracts facts. It does not need a homemade retry loop around every preference update. Engram records the work and drives it to commit. Partial failures recover cleanly. Commits stay atomic relative to the pipeline’s design.
GA hardening called out more durable pipelines as a production requirement for a reason. Memory drift is not only an LLM problem. It is also an orchestration problem. If updates arrive and vanish under load, agents look forgetful even when topic descriptions are perfect.
How does in-order processing protect reconciliation?
Durability alone is not enough if two writes for the same user race. Transform steps reconcile new facts against existing memories. Order matters. If “promoted to CEO” commits before “works as an engineer” finishes, you get different merge decisions than the reverse. Engram queues pipeline runs grouped by the scope ids you provide. Processing follows the order you added data for that scope.
You can still send many batches quickly. The low-latency API accepts them. The durable executor serializes work per scope so you do not manually lock users or conversations. Different scopes can proceed in parallel. The same user or conversation stays ordered. That split matches how agents actually behave under concurrency.
Buffers sit comfortably on this foundation. A run can pause in in_buffer while more pieces arrive, then continue without you tracking which partial memories are waiting. The workflow remembers where it stopped. Your application only keeps the run id if it cares to observe the outcome.
What should application code do on the hot path?
Knowing the backend is durable, the next question is how thin your client path can be. In most chat products, call client.memories.add, keep the conversation moving, and let search pick up committed memories later. Recent turns are already in the prompt. Waiting on every run adds latency you usually do not need.
Check the immediate add response for hard input errors. Use client.runs.wait or runs.get when tests, migrations, or admin tools need proof a specific run completed. Production traffic can stay eventually consistent. That is the intended split between durable background work and a responsive agent loop.
For high concurrency across users, AsyncEngramClient keeps your own request workers non-blocking while Engram’s workflows handle memory durability separately. Client async and pipeline durability solve different layers. Use both when load demands it.
How do you observe a durable run without owning the workflow?
You do not operate Temporal yourself when using managed Engram. You observe runs. A run moves through running, possibly in_buffer, then completed or failed. On completion, committed_operations lists creates, updates, and deletes. On failure, an error string explains the problem. The console Runs page shows the same trail for operators.
Here is a bronze foundry desk that sends two ordered notes for one hearth, waits only on the second run, and confirms search sees the reconciled state.
import os
from engram import EngramClient, HybridRetrieval
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
hearth = "smelter-hearth-4"
first = client.memories.add(
"Hearth 4 poured silicon bronze for gate valve blanks at 1040 C. "
"Skim the slag before the second lift.",
user_id=hearth,
group="default",
)
second = client.memories.add(
"Same pour: customer wants a smoother as-cast finish than lot SB-08. "
"Do not raise temperature further; adjust mold wash instead.",
user_id=hearth,
group="default",
)
# Both accepted quickly; durable execution keeps per-scope order.
status = client.runs.wait(second.run_id)
assert status.status == "completed"
hits = client.memories.search(
query="silicon bronze finish versus SB-08 and slag skim",
user_id=hearth,
group="default",
retrieval_config=HybridRetrieval(limit=5),
)
assert any(
"SB-08" in m.content
or "mold wash" in m.content.lower()
or "1040" in m.content
for m in hits
)
Both adds return before commit. Waiting on the later run is enough when you need a barrier. Engram’s scope-ordered queue is what keeps the second note from racing ahead of the first behind the scenes.
How should you design agents around durable pipelines?
Treat Engram as the system of record for long-term memory writes, not as a synchronous database round trip on every token. Emit events and conversations as soon as they happen. Let durable runs reconcile them. Search when you need history, not as a substitute for acknowledging the write.
Keep scope ids stable. In-order guarantees are only as useful as the keys you pass. If you mint a new conversation id every message, you lose the queue grouping that protects that thread. If you reuse a user id correctly, preference updates stay serialized for that person.
When debugging “the agent forgot,” separate model failure from workflow lag. Check run status and committed operations before rewriting topic descriptions. Durability makes lag observable. It also makes silent loss rare. That combination is why Engram can promise fire-and-forget without asking your service to become a workflow engine.
Our next chapter, What input data types does Engram accept?, returns to what you put on the wire. You will see how conversation, string, and pre-extracted inputs each enter the durable pipeline through their own extract steps.