How does asynchronous processing and run status work in Engram?

Asynchronous processing is how Weaviate Engram keeps memory writes off the agent hot path. When you call memories.add, Engram returns a run_id and a status such as running almost immediately. Extract, transform, and commit continue in the background through a durable pipeline. Memories become searchable when that run completes. This chapter explains why async is the default product shape, what each run status means, when to wait with client.runs.wait, how committed_operations reveal creates, updates, and deletes, and how to debug failures without blocking every chat turn.

Retrieval types decide how you read. Run status decides when a write has truly landed. Confusing those two layers is how teams invent “memory is flaky” bugs that are really timing bugs.

Why does add return before memories exist?

After you understand pipelines, the latency question arrives quickly. Extraction and reconciliation use LLMs and existing memory lookups. Doing that inside the HTTP request would make every “remember this” call slow. Engram’s design is fire-and-forget at the application layer. Accept the input. Start a run. Return. Let durable execution finish the work.

That matches how chat actually works. The newest messages are already in the model context. You rarely need the just-sent turn as a searchable memory before the next token. You need it later, in another session, or much deeper in the thread. Eventual consistency is not a compromise here. It is the correct default.

A successful add response means the pipeline started, not that commits finished. Read the returned run_id and initial status. Catch immediate validation errors on that response. Do not assume search will see the new facts in the same millisecond.

What do the four run statuses mean?

Knowing a run exists, the next question is how to interpret its state. running means the pipeline is actively processing. in_buffer means the run paused at a buffer step and is waiting for a trigger such as a count or timer. completed means all operations committed successfully. failed means processing hit an error.

Buffers make in_buffer a first-class status rather than a mystery hang. Continual-learning and rollup pipelines may wait until enough pieces arrive. Your poll loop should treat that as healthy waiting, not as a silent crash. Only failed needs error handling. Only completed guarantees committed_operations is the final story for that run.

The Engram console Runs page shows the same statuses for operators. You can filter recent runs, open a detail panel, and see which memories were created, updated, or deleted. Application code and human debugging share one model of progress.

When should you wait on a run?

Most production chat paths should not wait. Blocking on every add defeats the async pipeline and slows ingest under load. The SDK guidance is explicit. Treat add as fire-and-forget unless you genuinely need the outcome before continuing.

Wait when tests must assert that memories exist. Wait when an admin tool shows “what did this upload change.” Wait when a migration or backfill must prove completion before the next batch. Use client.runs.wait(run.run_id) for those barriers. Use a GET on the run id when you only need a snapshot. Prefer waiting on the specific run you care about rather than sleeping and hoping.

Client-side AsyncEngramClient is a separate concern. It keeps your own Python workers non-blocking while calling Engram. Pipeline async still applies either way. You can await a search without awaiting every preceding add.

What do committed operations tell you?

When a run completes, committed_operations lists the durable effects. created holds new memory ids. updated holds memories that were merged or refined. deleted holds memories removed because they were superseded. Each entry includes a memory_id and a committed_at timestamp.

That ledger is how you verify reconciliation without scraping search results. A preference change may update one memory and delete a duplicate rather than create a third conflicting fact. If your test only counts creates, you will misread healthy merges as failures. Read all three buckets.

On failure, inspect the error field. Invalid input formats and extraction failures surface there. Fix the payload or topic configuration, then retry with a new add. Do not invent a second write path that bypasses run tracking.

How does async processing look in a real write-then-verify path?

Here is a clock-tower workshop assistant that fires two notes, waits only when verification is required, and inspects committed operations before hybrid search.

import os
from engram import EngramClient, HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

tower = "clock-tower-gear-5"

# Hot path: accept work and keep moving.
accepted = client.memories.add(
    "Gear train 5 showed uneven wear on the third wheel after the Easter peal. "
    "Schedule a bushing check before the next full chime test.",
    user_id=tower,
    group="default",
)
print(accepted.run_id, accepted.status)

# Later verification path: wait only when you need a barrier.
verify = client.memories.add(
    "Same train: keep mainspring torque under the shop limit used for lot CT-17. "
    "Do not oil the escape wheel until bushings are confirmed.",
    user_id=tower,
    group="default",
)
status = client.runs.wait(verify.run_id)
print(status.status)
print(status.committed_operations)

assert status.status == "completed"
created = list(status.committed_operations.created or [])
updated = list(status.committed_operations.updated or [])
deleted = list(status.committed_operations.deleted or [])
assert created or updated or deleted

hits = client.memories.search(
    query="third wheel bushing and CT-17 torque before oiling",
    user_id=tower,
    group="default",
    retrieval_config=HybridRetrieval(limit=5),
)
assert any(
    "bushing" in m.content.lower()
    or "CT-17" in m.content
    or "escape" in m.content.lower()
    for m in hits
)

The first add never waits. The second waits because the example is proving a contract. In a live chat handler you would likely wait on neither, then search on a later turn when history matters. In CI you wait so assertions are deterministic.

How should you design agents around eventual consistency?

Separate “remember this” from “what do we already know.” Emit adds continuously. Search when composing prompts for older context. If a user asks about something said ten seconds ago, answer from the live transcript, not from Engram. If they ask about last week’s preference, search Engram.

When a memory seems missing, check run status before rewriting topics. A run still running or in_buffer is not a retrieval failure. A failed run is not a hybrid tuning problem. Async processing makes those distinctions visible. Use them. Engram’s run model turns background memory work into something you can observe, test, and trust.

Our next chapter, How does Engram handle deduplication and reconciliation?, looks inside what those committed updates and deletes are doing. You will see how new facts merge with existing memories instead of piling up forever.