How do you handle schema evolution in long-lived memory systems?

Long-lived agent memory outlasts the first schema you ship. Topics gain new meanings. Scope keys appear. Product language changes. Embeddings may need a fresh model years later. If you treat the store as a frozen prototype, every change becomes an emergency rewrite. If you treat it like a production database, change becomes routine. This chapter shows why memory schemas drift, which changes stay additive, when you must migrate beside the live path, how Weaviate supports safe collection evolution, and how Weaviate Engram keeps agent memory evolvable through topics, groups, and scoped writes without forcing you to redesign the chat loop every quarter.

Why Do Memory Schemas Drift After Launch?

Day-one memory design mirrors the first product story. You might store only user preferences. Six months later you need session summaries, tool outcomes, and domain facts that never fit that first bucket. Agents also rewrite what they believe. A preference reverses. A policy updates. A fact gains a qualifier. The stored shape must absorb those shifts without losing older useful records.

Drift also comes from the stack under the memories. Property indexes, vector settings, and retrieval filters evolve as query patterns mature. An embedding upgrade changes the coordinate space of every vector. Derived fields such as summaries go stale when the extraction prompt improves. Keeping the raw source text matters because derived forms can be rebuilt. Blind stores without a version story force readers to guess which fields exist. That guessing becomes fragile application code.

Once drift is inevitable, the practical question is which changes you can ship without stopping traffic.

Which Schema Changes Stay Additive, and Which Ones Force a Migration?

Additive change is the default safe path. Add a new optional property. Add a new topic description that captures a fresh category of facts. Add a new named vector beside the old one while dual-writing. Expand first. Dual-write while both shapes exist. Backfill history. Contract only after readers no longer need the old fields. That expand-contract rhythm lets old and new code coexist and keeps rollback possible.

Breaking change is different. Renaming a required field, changing a property’s data type, or removing a property that still has readers usually needs a new collection or a full rebuild path. In Weaviate, many collection settings are mutable after creation, but existing property definitions are not freely rewritten. You can add properties. You cannot casually reshape an existing property in place. Deleting a property type or changing its type means creating the destination schema and moving data. Collection aliases help here. Applications keep calling a stable alias while you build a new collection, copy or transform objects, then point the alias at the new target in one switch.

Late-added properties carry a subtle indexing caveat. Property indexes are built at import time. Objects that already existed may not appear in the new property’s index until you re-create or re-index the collection. Plan backfill when filters on the new field must cover the full history, not only new writes.

That database discipline maps cleanly onto Engram’s configuration model for agent memory.

How Does Weaviate Engram Make Memory Shape a Configuration Concern?

In Engram, the living “schema” for agents is mostly topics inside a group, not ad-hoc JSON keys invented in the client. A topic’s name and natural-language description tell the pipeline what to extract. Scoping decides whether a memory is project-wide, user-scoped, or further isolated by properties such as a reel id. Bounded topics keep at most one memory per scope, which is ideal for profiles and running summaries that must update in place rather than spawn duplicates.

When the product evolves, prefer configuration expansion over silent field invention. Add a topic for a new category of knowledge. Create a separate group when a use case needs its own pipeline and topic set. Keep older topics readable while new topics fill in. Search can filter by topic when a turn only needs the new category, or omit the filter when continuity across the whole group still matters. Reconciliation inside the pipeline already merges and rewrites overlapping facts, so content evolution often happens without a collection-level migration at all.

Application code should still stay disciplined. Pass the same user_id, group, and scope properties the topics expect. Wait on runs when a follow-up search must see the just-committed shape. Use hybrid retrieval so renamed product language still finds older wording during transition periods. Engram absorbs a large share of schema churn because extraction follows topic descriptions, not a brittle hand-written column list in every agent turn.

Here is a film-archive assistant evolving from simple reel notes into scoped shelf guidance. The write path stays stable even as the memory categories grow richer over time.

import os
from engram import EngramClient
from engram.types import HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
user_id = "archivist-lena"
group = "film_archive"

run = client.memories.add(
    "Reel bay 12 (reel-bay-12): nitrate safety print of Harbor Night must stay "
    "in cold vault C; do not move to ambient inspection benches without a "
    "humidity log. Projectionist notes from 2024 are superseded—use the 2026 "
    "inspection sheet only.",
    user_id=user_id,
    group=group,
    properties={"reel_id": "reel-bay-12"},
)
client.runs.wait(run.run_id)

results = client.memories.search(
    query="Where should the Harbor Night nitrate print be stored?",
    user_id=user_id,
    group=group,
    properties={"reel_id": "reel-bay-12"},
    retrieval_config=HybridRetrieval(limit=5),
)
for memory in results:
    print(memory.content)

Later you might add a topic for inspection-sheet lineage or a bounded vault-profile per reel. The client still calls memories.add and memories.search. The schema change lives in project configuration and scope properties, not in a one-off table rewrite inside the chat service.

Some changes still reach the vector store itself. Those need an explicit migration plan.

When Should You Rebuild Collections Instead of Extending Topics?

Rebuild when the coordinate space or the physical collection definition must change. A new embedding model is the classic case. Old vectors and new vectors are not comparable in one index without a dual-vector or dual-collection strategy. Named vectors or a second collection let you dual-write, backfill, switch search, then retire the old path. Collection aliases keep the application name stable during that cutover. The same pattern applies when an immutable Weaviate setting must change, or when a property type change would otherwise corrupt readers.

Preserve source text for anything derived. If you only keep embeddings, you cannot honestly rembed after a model upgrade. If you only keep summaries, you cannot regenerate them after a better prompt. Engram’s strength is that committed memories remain searchable text with vectors underneath. That gives operations a rebuild path when infrastructure schema must move, while day-to-day product schema mostly moves through topics and groups.

Version observability helps either path. Track which memories or collections still sit on an older shape. Measure backfill progress. Refuse silent “if field missing” thickets in readers. Prefer small tested upgrade steps. Schema evolution fails when it is improvised under an incident. It succeeds when expand, migrate, and contract are planned like any other production change.

Healthy indexes keep search fast. Evolving schemas keep search meaningful as the product itself changes. Prefer additive topic and property growth, use aliases and dual paths for breaking store changes, and let Weaviate Engram carry most agent-facing evolution in configuration rather than in emergency migrations. Our next chapter, How do you deploy multi-region memory for global agents?, asks what happens when that long-lived memory must live in more than one geography without confusing which region holds the truth.