How do you migrate data between memory system versions?

Disaster recovery plans for memory assume the stack version stays fixed while you restore. Planned migrations change the stack on purpose. Schema settings, embedding models, Weaviate minor versions, and Engram topic configurations all evolve. Many of those settings are immutable after a collection is created. You cannot quietly edit them in place. This chapter treats memory migration as a controlled cutover: backup first, build a parallel target, re-encode when vectors must change, validate with Weaviate Engram canaries, then switch traffic with a rollback path still warm.

Which Kind of Memory Migration Are You Actually Running?

Not every upgrade is the same job. A Weaviate server upgrade follows the documented minor-by-minor path with a full backup before you start. Schema or vectorizer changes usually require a new collection, because those settings define how objects are indexed. An embedding model swap is a data migration, not a config flip. Old vectors and new query vectors do not live in the same space. Engram group and topic changes alter what gets extracted going forward. They do not automatically rewrite history unless you reprocess source conversations.

Name the migration type in the runbook before you move a single object. The validation bar differs. A binary upgrade cares about cluster sync and restore integrity. An embedding rotation cares about retrieval quality on a golden query set. A topic redesign cares whether new memories land in the right scopes without polluting old ones.

Once the type is clear, the next question is how to keep agents online while the new version fills.

How Do You Migrate Without Turning the Cutover Into Downtime?

Weaviate collection aliases exist for this pattern. Applications talk to a stable alias name. You create a new collection with the desired schema or vectorizer. You copy or re-embed data into that collection while the old one still serves traffic. When checks pass, you update the alias to point at the new target. The switch is atomic. Rolling back means pointing the alias back, not restoring from cold storage under pressure.

For embedding changes, copy from source text, not from old vectors. Embeddings are not round-trippable. Re-run the new model on the original memory content. Dual-write new memories to both collections during backfill so the shadow stays current. Shadow-read a sample of production queries against both sides and compare overlap or labeled relevance before you flip the alias.

Managed Weaviate Engram hides much of the collection plumbing. You still own application-level cutover habits: pause risky bulk rewrites, pin embedding and topic versions in the runbook, and verify recall after the service-side change. Self-hosted stacks that expose Weaviate directly should use aliases explicitly so agent code never hard-codes a disposable collection name.

What Must Stay Compatible Across the Version Boundary?

After the mechanics, compatibility rules decide whether the migrate succeeds quietly or fails in production. Never search a new-model query vector against an old-model index. Scope every query to one embedding generation. Preserve scope properties during copy so user and project isolation survives. Carry provenance fields when you can: model version, topic set, and source hash help later audits when two memories disagree.

Upgrade Weaviate one minor version at a time to the latest patch of each step. Take a backup before the first hop. After major consensus changes, wait for cluster metadata to report synchronized before the next hop. Those steps prevent a memory migration from colliding with an unfinished platform migration.

Engram pipelines add another compatibility surface. If you change topics mid-flight, decide whether historical memories stay under the old semantics or get re-extracted from retained conversations. Mixing both without a flag produces silent drift that looks like a retrieval bug.

How Should Weaviate Engram Prove the New Version Is Safe?

Infrastructure green is not enough. Seed scoped canary memories before cutover. After the alias flip or Engram project switch, search those canaries with the same user, group, and properties. Confirm content and hybrid ranking still hold. Run a small golden pack of real support queries and compare hit quality to the pre-migration baseline. Keep the old collection or project readable through the rollback window.

Gate write reopening on that gate. Dual-write only while both sides are intentionally live. Once traffic is fully on the new version and the rollback window expires, retire the old collection to stop paying double storage. Document the retirement date beside the backup that covered the cutover.

Here is a map archive desk that records a pre-migration canary in Weaviate Engram and re-checks it after a version cutover. The same checks work whether the underlying move used collection aliases, a re-embed, or an Engram configuration bump.

import os
from engram import EngramClient, HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
group = "map_archive"
archivist = "archivist-nell"
drawer = "folio-drawer-c7"

# Record canary facts against the live version before migration
seed = client.memories.add(
    [
        {
            "role": "user",
            "content": (
                "Folio drawer C7 holds the 1891 harbor survey. Sheets must stay "
                "flat under glass weights, and requests for coastal soundings "
                "should prefer the linen-backed copy over the brittle paper set."
            ),
        },
        {
            "role": "assistant",
            "content": "Logged drawer C7 handling rule and preferred harbor survey copy.",
        },
    ],
    user_id=archivist,
    group=group,
    properties={"drawer_id": drawer, "cost_center": "cartography"},
)
client.runs.wait(seed.run_id)


def post_migration_gate(expected_model_tag: str) -> dict:
    """Run after alias flip / Engram version cutover; keep old side warm until ok."""
    hits = client.memories.search(
        query="Which harbor survey copy should we use from folio drawer C7?",
        user_id=archivist,
        group=group,
        properties={"drawer_id": drawer},
        retrieval_config=HybridRetrieval(limit=3),
    )
    joined = " ".join(m.content for m in hits).lower()
    content_ok = ("linen-backed" in joined) and ("1891" in joined)
    return {
        "drawer_id": drawer,
        "expected_model_tag": expected_model_tag,
        "hit_count": len(hits),
        "canary_ok": content_ok,
        "allow_cutover_complete": content_ok,
        "previews": [m.content[:110] for m in hits],
    }


# Example: after migrating to embedding/topic bundle "maps-v2"
gate = post_migration_gate("maps-v2")
print(gate)
if not gate["allow_cutover_complete"]:
    raise SystemExit("Migration gate failed: point alias back to the previous collection.")

The canary is intentionally dull. It catches missing scopes, broken hybrid search, and empty restores faster than a broad synthetic dump.

Which Habits Keep Migrations From Becoming Accidental Disasters?

Write the cutover as a dated change record. Include backup ID, source and target collection or Engram project, embedding pins, topic versions, dual-write start and stop times, gate results, and rollback owner. Rehearse once in staging with production-shaped volume. Measure backfill duration so the dual-write window is funded in the calendar, not guessed during the release.

Separate platform upgrades from embedding rotations when you can. Stacking both in one night multiplies failure modes. If you must combine them, finish the server upgrade and sync checks before you start re-embedding. Keep cost models honest. Re-embedding large memory corpora is a first-class line item, not a side effect.

Memory versions change safely when you treat the move like a database migration. Back up. Build parallel targets. Re-encode when vectors change. Validate with Engram canaries. Switch through aliases or equivalent cutover flags. Keep rollback warm until the gate stays green. High availability then protects that migrated store while it serves ordinary failures.

Our next chapter, How do you design high availability for memory services?, focuses on keeping that migrated memory tier online through node loss and rolling upgrades, using replication and failover patterns that match Engram’s read and write paths.