How do you design high availability for memory services?

Migrations put the right memory version in place. High availability keeps that version answering while nodes fail and upgrades roll. Agent products feel memory downtime as amnesia, not as a brief spinner. Users notice within a turn. This chapter designs HA for memory services on Weaviate: replication for node and zone loss, tunable consistency for read and write paths, rolling upgrades without cluster outages, and application habits that let Weaviate Engram keep searching when one replica is dark.

What Does High Availability Mean for a Memory Service?

High availability means the memory API stays useful through failures you expect inside a region. A pod restarts. An availability zone blips. A rolling upgrade takes one node offline. Search should continue. Fresh writes should either succeed or fail cleanly, never hang forever. That is different from disaster recovery, which rewinds after corruption or region loss using backups outside the cluster.

For agents, availability has a semantic face. An HTTP 200 with empty recall is still an outage. HA design therefore pairs infrastructure redundancy with health checks that prove Engram can still find scoped memories. Process liveness alone is not enough.

The core infrastructure lever is replication. Without copies, any node that holds a shard takes that memory offline when it dies. With copies, traffic moves to surviving replicas.

How Should You Shape Replication for Memory Workloads?

Weaviate stores object data with a configurable replication factor. A practical starting point is three replicas on at least three nodes, ideally across availability zones. Odd factors make quorum math clean. Cluster metadata such as collection definitions follows Raft, so schema changes survive a minority of node failures. Data replication is leaderless. Any healthy replica can serve, which removes a single primary as a bottleneck.

On Weaviate Cloud, enabling high availability at cluster creation wires multi-node topology and appropriate factors for you. Self-hosted clusters set replication on collections and ensure node count is at least the factor. If nodes are fewer than the factor, the cluster will not start correctly. Spread replicas so one node never holds every copy of the same tenant or shard.

Replication multiplies storage and memory. A factor of three roughly triples the footprint of the working set. Budget that cost explicitly. Teams that enable HA after they are already RAM-bound discover the bill the hard way. Quantization and right-sized indexes still matter on each replica.

How Do Consistency Choices Affect Agent-Facing Behavior?

After topology, consistency levels decide how strict each request is. Weaviate lets reads and writes wait for ONE, QUORUM, or ALL replicas. Quorum means a majority of the replication factor. Quorum on both sides is a balanced default for memory products that need both freshness and availability. Write ALL with read ONE favors durable commits and fast reads when stale answers are unacceptable on write. Write ONE with read ALL favors ingest speed and stricter reads.

Memory agents usually prefer search availability during partial failure. A slightly stale preference is often better than a hard error on every turn. Critical profile updates may still deserve stronger write acknowledgment. Encode that split in the application. Hot-path memories.search should tolerate replica loss. Administrative commits can wait longer.

Expect brief divergence when a node is down during writes. Async and read repairs bring replicas back in line. Your Engram canaries after maintenance should confirm that known facts are visible again before you declare the rolling event finished.

How Do Application Clients Stay Resilient on Top of HA?

Cluster HA does not remove the need for client discipline. Time out and retry idempotent searches. Prefer AsyncEngramClient when many scopes must be queried under load, so one slow replica path does not serialize the whole turn. Keep writes fire-and-forget on the chat path. Pipeline durability in Engram already queues extract and transform work. Do not block the user on runs.wait unless you are in a drill or batch job.

Place load balancers and multi-AZ networking in front of the Weaviate nodes that back Engram. Health checks should exercise a cheap scoped search, not only TCP. During rolling upgrades, Weaviate experiments with replication factor three have shown query success continuing while individual pods restart. Without replication, a large share of requests fail for the length of the restart window.

Here is an archery range desk that keeps lane memories available through concurrent Engram searches. The pattern matches how an HA cluster absorbs parallel agent traffic while one replica is restarting.

import os
import asyncio
from engram import AsyncEngramClient, HybridRetrieval

client = AsyncEngramClient(api_key=os.environ["ENGRAM_API_KEY"])
group = "archery_range"
coach = "coach-priya"
lanes = ["lane-score-9", "lane-score-10", "lane-score-11"]


async def seed_lane(lane_id: str) -> str:
    run = await client.memories.add(
        [
            {
                "role": "user",
                "content": (
                    f"{lane_id}: keep the clicker gap at 4mm for recurve warm-ups, "
                    "and pull athletes off the line if crosswinds exceed 18 km/h."
                ),
            },
            {
                "role": "assistant",
                "content": "Logged clicker gap and wind safety threshold for this lane.",
            },
        ],
        user_id=coach,
        group=group,
        properties={"lane_id": lane_id, "cost_center": "range_ops"},
    )
    return run.run_id


async def search_lane(lane_id: str) -> dict:
    hits = await client.memories.search(
        query="What clicker gap and wind limit apply on this scoring lane?",
        user_id=coach,
        group=group,
        properties={"lane_id": lane_id},
        retrieval_config=HybridRetrieval(limit=3),
    )
    text = " ".join(m.content for m in hits).lower()
    return {
        "lane_id": lane_id,
        "hit_count": len(hits),
        "healthy": ("4mm" in text) and ("18" in text),
    }


async def ha_readiness_probe() -> dict:
    # Seed once in staging; in production probes usually search only
    await asyncio.gather(*(seed_lane(l) for l in lanes))
    # Give pipelines a moment in drills; hot paths skip waits
    await asyncio.sleep(2)
    results = await asyncio.gather(*(search_lane(l) for l in lanes))
    ok = all(r["healthy"] for r in results)
    return {"group": group, "all_lanes_ok": ok, "results": results}


print(asyncio.run(ha_readiness_probe()))

Run the probe after node drains and during upgrade windows. If any lane fails while the cluster claims to be up, treat it as a memory HA incident, not a cosmetic flake.

Which Operational Habits Keep HA Real?

Practice node kills and rolling upgrades on a schedule. Confirm Engram probes stay green. Watch replica health, repair lag, and search p95 beside ordinary QPS. Annotate dashboards when replication factor or consistency defaults change. Keep backups even with HA. Replication does not rewind a bad delete that reached every copy.

Size for peak on the surviving capacity, not the full healthy set. If you need two nodes to carry the load when one is down, provision three that can each take half-plus headroom. Memory services that only fit when every replica is perfect are not highly available. They are fragile clusters with extra bills.

High availability for memory is replication across failure domains, consistency choices that match agent tolerance, clients that search concurrently and fail soft, and probes that prove recall. Design for node loss and rolling upgrades first. Leave true region disasters to the recovery playbooks you already wrote. Then protect the write path from stampeding itself.

Our next chapter, How do you rate-limit and throttle memory write pipelines?, covers how to keep Engram ingest and extract workers healthy when agents try to write faster than the cluster should accept.