How do you scale memory systems for many concurrent agents?

Concurrency is where memory systems stop looking like notebooks and start looking like infrastructure. Ten agents writing politely is a demo. Two hundred agents writing during a rush-hour surge is a product. Scaling memory for many concurrent agents means keeping writes cheap to accept, keeping scopes isolated so one station does not poison another, and keeping search fast enough that specialists still act with fresh context. This chapter names the bottlenecks that appear under load, explains how partitioning and async clients absorb spikes, shows how Weaviate Engram’s low-latency adds and per-scope ordering help, and outlines operational habits that keep a swarm usable when every agent wakes at once.

What Breaks First When Many Agents Share One Memory Store?

The first failure is rarely disk space. It is contention of meaning. Agents dump overlapping observations into one unscoped pool. Search returns a mash of unrelated stations. Latency climbs because every query scans a swollen shared namespace. Operators add replicas and still get confused answers. Throughput was never the only limit. Isolation was.

The second failure is client blocking. A synchronous memory client waits on each add or search while dozens of peer agents sit idle. Inference already costs seconds. Serial memory calls multiply that cost across the fleet. Under a surge, the orchestrator looks stuck even when Engram accepted the writes quickly.

The third failure is waiting on every run. Critical handoffs need runs.wait. Mass sensor chatter does not. If every concurrent agent blocks until commit, you turn an async pipeline into a traffic jam. Scale requires knowing which writes must be durable before the next act, and which can finish in the background.

How Do You Partition Work So Concurrency Stays Safe?

Once the failure modes are clear, partitioning is the main design lever. Give each concurrent unit a scope that matches the real world boundary. In a bike-share rebalancing surge, that unit is often a station id. Station agents write with properties={"station_id": ...}. City-wide playbooks live in a separate group. A dispatcher searching one station never inherits gossip from a station across town.

User ids can represent agent roles or rider threads when privacy matters. Groups separate use cases so rebalancing scratch never shares topics with billing support. Engram isolates groups with multi-tenancy. That keeps topic names reusable and keeps noisy workloads from colliding in one bag.

Partitioning also helps ordering. Engram processes raw data in order within a scope. Concurrent adds across different station ids proceed without waiting on each other. Concurrent adds for the same station stay coherent. That is how you get both parallelism and sanity.

How Does Weaviate Engram Absorb a Concurrent Agent Spike?

Partitioning still needs an API posture that matches the load. Weaviate Engram accepts memory writes asynchronously. memories.add returns a run id quickly while extraction continues in the background. For many concurrent agents, use AsyncEngramClient and gather searches or adds so the orchestrator does not serialize on network round trips.

Search with hybrid retrieval and tight scopes. Limit result counts. Wide unscoped searches during a surge are how one hot group becomes everyone’s bottleneck. Prefer station-scoped queries for local agents. Reserve broader searches for a smaller set of coordinator agents.

Here is a rush-hour bike-share rebalancing wave where many station agents write and search concurrently:

import asyncio
import os
from engram import AsyncEngramClient, HybridRetrieval

client = AsyncEngramClient(api_key=os.environ["ENGRAM_API_KEY"])
wave = "rebalance-wave-7"
stations = ["st-14th-pine", "st-market-5", "st-harbor-2", "st-campus-gate"]

async def station_tick(station_id: str, note: str):
    run = await client.memories.add(
        note,
        user_id=f"agent-{station_id}",
        group="bikeshare_rebalance",
        properties={"wave_id": wave, "station_id": station_id},
    )
    # Fire-and-forget for telemetry-style notes; do not wait under surge
    ctx = await client.memories.search(
        query="dock fullness van need overflow risk",
        user_id=f"agent-{station_id}",
        group="bikeshare_rebalance",
        properties={"wave_id": wave, "station_id": station_id},
        retrieval_config=HybridRetrieval(limit=4),
    )
    return station_id, run.run_id, [m.content for m in ctx]

async def surge():
    notes = {
        "st-14th-pine": "Station st-14th-pine is at 92 percent full docks. Request van pickup within 20 minutes.",
        "st-market-5": "Station st-market-5 has only three usable bikes. Prioritize drop-off over pickup.",
        "st-harbor-2": "Station st-harbor-2 reports two docks jammed. Mark as partial capacity.",
        "st-campus-gate": "Station st-campus-gate stable at 40 percent. No van needed this wave.",
    }
    tasks = [station_tick(sid, notes[sid]) for sid in stations]
    return await asyncio.gather(*tasks)

results = asyncio.run(surge())

Each station keeps its own scope. The wave id lets a coordinator later review the whole surge without mixing unrelated days. Concurrent agents scale because they mostly do not contend on the same scope key.

When Should Coordinators Wait, and When Should They Fan Out?

The station loop above favors throughput. Coordinators still need stronger sync at decision points. Before dispatching vans, a coordinator may wait on the run ids for stations it is about to serve. That is a narrow wait set, not a fleet-wide barrier. Scale dies when every agent waits on everyone else’s commits.

Fan-out reads the same way. Search stations in parallel with the async client. Merge results in the orchestrator. Do not ask Engram for one giant unscoped query that returns every station note in the city. Your application can union four tight searches faster than one muddy search can stay relevant.

Promote only durable lessons into a project-wide playbook group. Leave ephemeral fullness readings in the wave-scoped station properties. That keeps the hot path small. Small hot paths are what survive concurrency.

What Operational Habits Keep Concurrent Memory Healthy?

After the API patterns settle, operations decide whether the system stays fast next month. Cap how often agents write. Not every sensor tick deserves a memory. Write when a human or downstream agent would act differently because of the fact. Otherwise you pay extraction cost for noise.

Watch scope cardinality. Millions of one-off property values can be correct and still painful to reason about. Prefer stable station ids and wave ids over random uuids on every message. Keep group count intentional. A new group per agent is usually a design smell. A group per use case is usually right.

Finally, load-test the resume and search paths, not only adds. Concurrent agents fail in production when search latency spikes under write load. Measure p95 search with realistic scopes during a synthetic surge. If those numbers drift, tighten scopes and reduce unscoped coordinator queries before you add more agents.

Our next chapter, What is memory in human-in-the-loop agent systems?, turns from machine concurrency to the harder mix of agents and people, where memory must respect approvals, overrides, and human-authored corrections.