How do you A/B test memory configurations?

Pollution probes catch bad facts after they land. A/B tests decide which memory configuration produces better outcomes before you promote it everywhere. Memory systems have many knobs: retrieval type, result limit, topic filters, group routing, and write-side topic descriptions. Changing several at once creates stories, not evidence. This chapter shows how to A/B test Weaviate Engram configurations with one variable per experiment, how to score variants on the same probe set, and how to promote a winner without mistaking noise for progress.

Why Do Memory Tweaks Need Controlled Experiments?

Teams often ship a new hybrid limit after one impressive demo turn. The next week, keyword-heavy questions regress. Another team swaps vector-only search because a semantic paraphrase looked better in a notebook. Both changes can be right for some queries and wrong for others. Without a fixed evaluation set and a single changed variable, you cannot tell which edit caused which result.

RAG experimentation practice is clear on this point. Change one component at a time. Hold the rest to the current best config. Offline eval comes before canary traffic. Memory work follows the same discipline. Engram makes the retrieval side easy to vary through retrieval_config, topics, and group. The hard part is experimental hygiene, not API surface.

A good memory A/B answers a narrow question. Does hybrid with limit four beat hybrid with limit eight on sail-spec probes? Does topic-filtered search beat unfiltered search for fastener questions? Those are decideable. “Make memory better” is not.

Which Engram Knobs Are Fair to Put in an A/B?

Once the question is narrow, pick knobs that your application actually controls at call time. Retrieval type is the first family: HybridRetrieval, VectorRetrieval, and BM25Retrieval, each with a limit. Hybrid is the usual default. Vector favors paraphrase. BM25 favors exact part numbers and material codes. Limits change both prompt size and, on approximate indexes, how deep the search may explore.

Topic filters are the second family. Searching only a constraints topic can raise precision and hide useful preferences. Groups are the third family when you maintain separate pipelines for distinct use cases. Write-side topic descriptions and bounded topics are slower experiments. They change what gets stored, so you need matched corpora and time for pipelines to settle before you compare search quality.

Do not A/B user identity or scope properties as if they were retrieval preferences. Those are tenancy and correctness controls. Mixing them into a quality experiment will produce unsafe conclusions even if the metrics look neat.

How Should You Run a Retrieval Config Bakeoff With Weaviate Engram?

Start offline. Build a small golden set of queries with expected memory contents or acceptance notes. For each query, run variant A and variant B with identical user_id, group, and properties. Score hit presence, rank of the critical memory, contradiction rate, and search latency. Use paired comparisons on the same queries so variance stays honest.

Keep generation fixed while you test retrieval. Otherwise you will attribute a prompt rewrite to a memory change. After a retrieval winner is stable offline, canary it on a slice of live traffic. Watch correction rate, empty useful-context rate, and p95 search time. Promote only when both quality and latency budgets hold.

Here is a sail-loft rigger desk that compares two hybrid limits on the same panel probes without touching writes:

import os
import time
from engram import EngramClient, HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
rigger = "rigger-jon"
group = "sail_loft_ops"
panel = "sail-panel-14"

probes = [
    {
        "query": "What cloth weight is specced for panel 14 mainsail patches?",
        "must_include": "4.5 oz",
    },
    {
        "query": "Which thread batch is approved for the panel 14 leach repairs?",
        "must_include": "bonded polyester 138",
    },
    {
        "query": "Is panel 14 cleared for saltwater delivery this Friday?",
        "must_include": "cleared",
    },
]

variants = {
    "hybrid_limit_3": HybridRetrieval(limit=3),
    "hybrid_limit_8": HybridRetrieval(limit=8),
}

report = {}
for name, retrieval in variants.items():
    scores = []
    latencies = []
    for probe in probes:
        t0 = time.perf_counter()
        hits = client.memories.search(
            query=probe["query"],
            user_id=rigger,
            group=group,
            properties={"panel_id": panel},
            retrieval_config=retrieval,
        )
        latencies.append((time.perf_counter() - t0) * 1000)
        blob = " ".join(m.content.lower() for m in hits)
        scores.append(1.0 if probe["must_include"].lower() in blob else 0.0)
    report[name] = {
        "hit_rate": sum(scores) / len(scores),
        "avg_search_ms": round(sum(latencies) / len(latencies), 1),
        "n_probes": len(probes),
    }

print(report)
# Promote only if hit_rate rises without breaking the latency budget for the loft desk.

This is a miniature offline bakeoff. Production tests need more probes and a significance check. The shape stays the same: one changed retrieval config, shared scopes, scored on the same questions.

What Mistakes Make Memory A/B Results Untrustworthy?

Changing limit and retrieval type together is the classic confound. So is comparing a fresh group with different topics against an old group that already holds richer history. Equalize the memory store when the experiment is about search. When the experiment is about write configuration, rebuild both sides from the same transcript replay before scoring.

Another trap is judging only average score. A variant can win overall while failing the three safety-critical probes. Report per-slice results for fasteners, schedules, and preferences separately. Watch for pollution regressions that look like “more context” but inject contradictory setpoints.

Sample size matters. Retrieval parameter tweaks often need hundreds of queries before a small effect is trustworthy. Prompt and generation changes need even more. Do not crown a winner from a dozen hand-picked chats.

How Do You Promote a Winner Without Freezing Iteration?

Write the winning config into a single source of truth your agent loads at runtime. Treat it like the current baseline for the next experiment. The next test should change one new variable against that baseline, not against an ancient default. That sequential promotion path avoids local maxima built on stale controls.

Keep a shadow path during canary. Log what the losing variant would have retrieved without injecting it into the user prompt. Shadow scoring is cheap insurance when the metric is retrieval quality. Full answer interleaving is harder for memory-augmented chat, because users see one reply, not two merged lists.

A/B testing turns Engram configuration from folklore into a ledger. Isolate one knob. Score paired probes. Canary with latency in view. Promote the winner, then run the next honest experiment.

Our next chapter, How do you human-evaluate agent personalization quality?, covers the judgments automated hit rates cannot make alone, and how people score whether memory-backed personalization actually feels right.