Short answer: Debug one Engram run at a time—read status, error text, and the full committed creates/updates/deletes—rather than guessing from chat logs alone.
Observability gives you join keys; run-level debugging teaches you to read one pipeline execution when those keys point at trouble. Every memories.add becomes an asynchronous run with status, timestamps, optional error text, and—on success—committed creates, updates, and deletes. Chat logs show what people said, not what Engram kept: two identical-looking turns can create three memories or none. This chapter classifies running, in_buffer, completed, and failed; explains empty commits versus real mutations (including topic filters that drop facts); and shows how to reproduce a bad write with the same user_id, group, and properties. Always read the full operations object before filing a pollution bug—preference changes may update or delete rather than only create. Keep production turns async; use sampled waiters in CI and on-demand waiters in the runbook.
Observability gives you the join keys. Run-level debugging teaches you how to read one pipeline execution when those keys point at trouble. Weaviate Engram processes every memories.add as an asynchronous run with a status, timestamps, optional error text, and — on success — a precise list of committed creates, updates, and deletes. This chapter shows how to classify running, in_buffer, completed, and failed, how to interpret empty commits versus real mutations, and how to reproduce a bad write in a debugger without blocking ordinary studio-floor turns.
Why Is a Single Run the Right Unit of Memory Debugging?
Chat logs show what people said. They do not show what Engram decided to keep. Two turns can look identical in the UI while one run creates three memories and the other creates none. Debugging at the conversation layer alone invites superstition. Debugging at the run layer gives you a closed artifact: input type, scope, status, and committed operations.
Async systems fail quietly unless something ties the background work back to the request. Agent teams already know this pain from queue workers and tool fans. Memory is the same class of problem. The run_id returned by memories.add is that tie. Store it beside your session id. When a gaffer complains that annealer notes vanished overnight, start from the run, not from a fresh guess about embeddings.
Engram’s console already lists runs by status for the same reason. The API path is runs.wait or a status fetch for the same id. Use those tools when you are testing, replaying an incident, or proving a fix. Leave them off the interactive hot path so debugging discipline does not become a latency tax.
What Do the Four Run Statuses Actually Mean in Practice?
Once you open a run, the first field to trust is status. running means the pipeline is still working. Short waits are normal. Long stuck running states need timestamps and a second poll before you declare a hang. completed means every planned commit finished. It does not guarantee that new memories were created. Topics and transforms may correctly decide there was nothing durable to store.
failed is the clear alarm. The error field explains the fault, such as an invalid input format during extraction. Fix the payload shape, the content type, or the upstream formatter, then retry with a new add. Do not keep searching for a memory that a failed run never committed.
in_buffer is the status teams misread most often. A buffer step pauses on purpose until a count or time trigger fires. Aggregation pipelines use this to batch daily summaries. Treating buffer pause as failure will send you chasing healthy design. Record expected buffer windows next to the pipeline config. Teach on-call that paused can be correct.
How Do You Read Committed Operations When a Run Completes?
Status alone is not enough after completed. Open committed_operations. Creates list brand-new memory ids. Updates list merges and refinements. Deletes list supersessions. Each entry carries a committed_at timestamp. That list is the write-side ground truth for the run.
Empty created, updated, and deleted arrays after a completed run usually mean the extract or transform stage found nothing to change. That can be correct for chitchat. It can also mean topic configuration filtered out the facts you care about. Distinguish those cases by replaying with richer input or a pre-extracted fact when you need a hard proof that commit still works.
Updates and deletes matter during preference changes. If a kiln schedule note replaces an older one, you should see an update or a delete paired with a create, depending on pipeline design. Searching only for creates will make legitimate reconciliations look like data loss. Always read the full operations object before filing a pollution bug.
How Should You Reproduce a Bad Run With Weaviate Engram?
Reproduction starts from the logged run_id and the original scoped add. Rebuild the same user_id, group, and properties. Prefer the same message payload you stored in your conversation archive. Then wait on the new run in a debugger or canary job. Compare status, error, and committed operations to the incident run.
Immediate errors can appear on the add response itself. Catch those before you ever poll. Pipeline errors appear later on the run status. Both layers matter. A clean add with a later failure is still a broken write. A rejected add never produced durable work to search for.
Here is a glass-studio annealer desk debugger that submits a temperature note, waits for the run, and classifies the outcome:
import os
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
gaffer = "gaffer-nora"
group = "glass_studio"
bay = "annealer-bay-2"
run = client.memories.add(
[
{
"role": "user",
"content": "Bay 2 annealer hold is 920F for the thick vessel batch. Do not open early.",
},
{
"role": "assistant",
"content": "Recorded annealer hold for bay 2 thick vessels.",
},
],
user_id=gaffer,
group=group,
properties={"bay_id": bay, "debug_case": "annealer-hold-920"},
)
# Debugging / canary only — not the shop-floor chat path
status = client.runs.wait(run.run_id)
ops = getattr(status, "committed_operations", None) or {}
created = ops.get("created") or []
updated = ops.get("updated") or []
deleted = ops.get("deleted") or []
error = getattr(status, "error", None)
if status.status == "failed":
verdict = "pipeline_failed"
elif status.status == "in_buffer":
verdict = "paused_in_buffer_expected_or_stuck"
elif status.status == "completed" and not (created or updated or deleted):
verdict = "completed_noop_check_topics_or_input"
elif status.status == "completed":
verdict = "completed_with_mutations"
else:
verdict = f"unexpected_status:{status.status}"
print({
"run_id": run.run_id,
"status": status.status,
"verdict": verdict,
"error": error,
"created_count": len(created),
"updated_count": len(updated),
"deleted_count": len(deleted),
"created_ids": [
(c.memory_id if hasattr(c, "memory_id") else c.get("memory_id"))
for c in created
],
})
The verdict string is what you paste into the incident ticket. It forces a choice among failure, buffer pause, empty commit, and real mutation before anyone debates retrieval quality.
Which Fixes Belong at the Run Layer Versus the Search Layer?
After classification, pick the smallest fix. Failed runs with format errors need payload validation before add. Empty completed runs with missing durable facts need topic or extract tuning. Buffer pauses that never resume need trigger configuration review. Completed mutations that still miss in search are not run bugs. Those move to scope mismatch, ranking, or limit issues covered by observability traces.
Resist the urge to call runs.wait on every production turn after one bad night. That freezes users for a class of problems that should be handled with better logging and canaries. Keep a sampled waiter in CI. Keep an on-demand waiter in your runbook. Keep the shop floor async.
Run-level debugging turns Engram’s asynchronous pipeline into something you can interrogate. Read status honestly. Read committed operations completely. Reproduce with the same scopes. Then fix the step that actually broke.
Our next chapter, How do you detect memory pollution and drift in production?, asks what happens when runs succeed but the stored facts slowly go wrong, and how to spot pollution and drift before agents trust bad memory.