How do you trace a memory from extraction to retrieval?

Cost models tell you what memory spends. Observability tells you what a single memory did on the way from raw chat to a searchable fact. Asynchronous pipelines hide work by design. That is good for latency and bad for guesswork when a setter’s grade note never shows up in search. This chapter traces one Engram run from memories.add through extract, transform, and commit, then into a later memories.search. It shows which identifiers to log, how runs.wait and committed operations close the write side of the story, and how to join that story to retrieval without turning every user turn into a blocking poll.

Why Does Memory Fail Quietly Without a Trace?

Agents fail in public. Memory often fails in private. The reply still streams. The write returns quickly with a run id. Minutes later a search misses the fact everyone thought was stored. Without a correlation id, the team argues about extraction quality, scope filters, eventual consistency, or a bad query. All four can look the same from the chat UI.

Observability is the discipline of making that path inspectable. You need a write-side story and a read-side story. The write side follows one pipeline run. The read side follows one search that should have returned the committed memory. When both carry the same user, group, and property scopes, you can prove whether the memory was never committed, committed under a different scope, or committed and simply not ranked into the limit.

Industry practice is moving the same way. OpenTelemetry GenAI conventions already describe spans for creating and searching memory. The names differ by toolkit. The idea does not. Treat Engram calls as first-class spans in the agent trace, not as anonymous HTTP noise.

What Is the Path From Extraction to Retrieval Inside Engram?

Once you accept that silence is the default failure mode, you need a mental map of the pipeline. Weaviate Engram stores memories through an asynchronous DAG. A conversation or string enters at an extract step. Transform steps refine, merge, and reconcile against existing memories. Commit persists create, update, and delete operations. Some pipelines pause in a buffer until a count or time trigger fires. That pause is a real status, not a hang.

Each memories.add call creates a run. The API returns a run_id immediately while status is typically running. Later the run becomes completed, failed, or in_buffer. On completion, committed_operations lists which memory ids were created, updated, or deleted and when. That list is the ground truth for the write side. Guessing from chat text alone is not enough.

Retrieval is a separate hop. Search does not automatically know which run produced a hit. Your application must join them with scopes and timestamps. Store the run id next to the session id in your own logs. When a later search misses, look up the run first. If the run failed, fix extraction input. If it completed with creates, inspect scopes and the search query. If it is still in a buffer, wait for the buffer trigger or redesign the pipeline for that class of facts.

How Do You Instrument One End-to-End Trace With Weaviate Engram?

The map is useless unless your code emits the join keys. Log the agent request id, the Engram run_id, user_id, group, and the same properties you will use at search time. Time the add call separately from any wait. Time the search call on the hot path. Keep waits out of ordinary user turns. Use them in tests, canaries, and incident replay.

For a controlled trace, wait once after add, print committed memory ids, then search with the identical scope. That is how you prove the pipeline and the index agree. In production you still fire-and-forget most writes. You keep the run id so an on-call engineer can reconstruct the same proof later.

Here is a climbing-gym route-setting desk that traces one grade-note write through commit and into a scoped search:

import os
import time
from engram import EngramClient, HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
setter = "setter-kai"
group = "route_setting"
wall = "route-wall-b9"
trace_id = "desk-trace-8814"

t_add = time.perf_counter()
run = client.memories.add(
    [
        {
            "role": "user",
            "content": "Wall B9: the red overhang is now 5.11a. Holds were reset after the Friday open.",
        },
        {
            "role": "assistant",
            "content": "Logged the B9 red overhang grade update for setter Kai.",
        },
    ],
    user_id=setter,
    group=group,
    properties={"wall_id": wall, "trace_id": trace_id},
)
add_ms = (time.perf_counter() - t_add) * 1000

# Observability path only: confirm commit, then verify retrieval
status = client.runs.wait(run.run_id)
created = []
if getattr(status, "committed_operations", None):
    created = [
        op.memory_id if hasattr(op, "memory_id") else op.get("memory_id")
        for op in (status.committed_operations.get("created") or [])
    ]

t_search = time.perf_counter()
hits = client.memories.search(
    query="What is the current grade for the red overhang on wall B9?",
    user_id=setter,
    group=group,
    properties={"wall_id": wall},
    retrieval_config=HybridRetrieval(limit=5),
)
search_ms = (time.perf_counter() - t_search) * 1000

print({
    "trace_id": trace_id,
    "run_id": run.run_id,
    "run_status": status.status,
    "created_memory_ids": created,
    "add_ms": round(add_ms, 1),
    "search_ms": round(search_ms, 1),
    "hit_count": len(hits),
    "hit_preview": [m.content[:120] for m in hits[:3]],
})

This script is a canary, not the chat hot path. Ordinary turns should log run_id and move on. The canary proves extract-to-retrieval still works for the wall scope before Friday’s open-night traffic.

Which Signals Separate Pipeline Bugs From Retrieval Bugs?

After instrumentation exists, incidents become classification problems. A failed run with an error string is a pipeline bug. A completed run with empty committed operations may mean the topics filtered everything out. That is configuration, not network failure. A completed run with creates that never appear under the expected user_id and properties is almost always a scope mismatch between write and search.

Retrieval bugs look different. Hits return, but the wrong memory ranks first. Or the right memory exists under a broader search and disappears when you tighten properties. Or search latency spikes while run completion times stay flat. Those point to index load, limits, or query wording, not to extract.

Buffer states deserve their own playbook. A run stuck in_buffer is waiting by design. Operators who treat it as failure will “fix” healthy aggregation pipelines. Log buffer expectations beside the run id. Teach on-call that paused is not the same as failed.

How Do You Keep Traces Useful Without Flooding Logs or Leaking Data?

Full message bodies in every span create privacy risk and storage cost. Prefer ids, scopes, counts, statuses, and latencies by default. Sample content previews in staging. Redact member names in shared gyms when policies require it. Keep raw chat in your existing conversation store keyed by the same trace id.

Cardinality matters too. High-volume agents should not emit unbounded custom labels for every wall hold color. Stick to stable dimensions: group, wall id, run status, operation counts. Aggregate p95 add latency, p95 search latency, failure rate, and empty-commit rate per group. Drill into a single run_id only when an alert fires.

Observability turns Engram’s async design from a black box into a story you can audit. Capture the run. Confirm commit when you must. Search with the same scopes. Then you can fix the real stage instead of rewriting the whole memory stack.

Our next chapter, How do you do run-level debugging in asynchronous memory pipelines?, goes deeper on failed and buffered runs, and shows how to read committed operations when a single pipeline execution misbehaves.