What are memory maintenance jobs?

Memory maintenance jobs are how preference drift policies, prune rules, and merge heuristics actually run when nobody is chatting. Per-turn writes keep Engram fresh for the active user. Scheduled jobs sweep the quiet scopes. They search within a user_id and property boundary, merge near-duplicates, delete true junk, and optionally roll episodic notes into a daily summary. Weaviate Engram already does part of this inside pipelines. Buffers can flush every twenty-four hours into an aggregate transform. Your application still owns the cron that walks tenants, applies house rules, and calls memories.add, memories.search, and memories.delete with the right scopes.

This chapter covers what belongs in a maintenance job versus the write path, how Engram buffers and run status fit an offline sweeper, how to structure cleanup and merging safely, and a concrete nightly job sketch. Staleness detection for long-running deployments builds on these jobs. First you need reliable scheduled work.

Why can’t chat-time transforms replace scheduled maintenance?

After drift detectors and reconsolidation, it is tempting to declare the store self-healing. Chat only touches memories that were retrieved. Quiet users accumulate duplicates nobody opens. Bulk imports leave siblings that transform never saw together. Seasonal facts expire without a turn that mentions them. Maintenance jobs exist for the work that does not fit in a single request’s latency budget.

Production memory systems commonly run nightly lifecycle workflows. Score, consolidate, prune, emit metrics. Engram’s own design philosophy matches that split. Interactive adds should stay fire-and-forget. Heavy merge and expiry run asynchronously. Pipeline buffers are the in-service version of a schedule. Application cron is the version you control when pipeline DAGs are fixed or when rules span many users.

Keep chat-path transforms for immediate corrections. Keep jobs for fleet hygiene. Do not block a user reply on a full-scope duplicate scan.

What should a maintenance job do, in what order?

Once you accept offline work, order matters. A useful sequence is inventory, merge, drift apply, prune, verify. Inventory searches representative queries per scope and records counts by topic and age. Merge sends a synthesized canonical string through memories.add so Engram transform can reconcile, then deletes leftover absolute duplicates you still find. Drift apply runs your short-versus-long preference detector and writes corrections. Prune deletes or archives memories that fail age and relevance rules. Verify re-searches and logs remaining conflict pairs.

Prefer merge-before-prune. Pruning first can delete one side of a pair you meant to combine. Prefer archive-before-hard-delete for anything that might be audited. Engram delete is permanent. Cold copies live in your own store.

Always partition by tenant. Loop users and property scopes explicitly. A global unscoped sweep is how you mix customers. Pass user_id and group on every get and delete. Property filters on every search that should stay soft-isolated.

How do Engram buffers and runs participate in scheduled work?

Knowing the job steps, reuse Engram machinery where you can. A pipeline shaped as extract, transform, commit, buffer, transform, commit can produce daily rollups without your cron writing the summary text. The buffer trigger is the schedule. Your job then only needs to ensure yesterday’s data was added, then inspect completed runs.

When you drive maintenance yourself, treat each correction as a normal add. Call runs.wait when the next step in the same job depends on the merge being searchable. Read committed_operations to learn which ids were created, updated, or deleted. Failed runs should abort the prune phase for that scope so you do not delete originals after a failed consolidate.

Mutex the job. Overlapping nightlies on the same scope cause double deletes and races with live traffic. Stagger large fleets. Start with high-churn scopes, then the long tail.

What does a nightly cleanup-and-merge job look like?

Imagine a cooperage floor agent on cooperage-hoop-drive-bench-4. Overnight the job merges duplicate hoop-tension notes and removes a superseded absolute setpoint.

import os
import re
from datetime import datetime, timezone, timedelta
from engram import EngramClient, HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
cooper = "cooper-nell-brady"
bench = {"bench_id": "cooperage-hoop-drive-bench-4"}
STALE = timedelta(days=120)
archive = []

def parse_ts(value: str) -> datetime:
    return datetime.fromisoformat(value.replace("Z", "+00:00"))

def inventory(query: str = "hoop tension oak barrel"):
    return client.memories.search(
        query,
        user_id=cooper,
        group="personalization",
        properties=bench,
        retrieval_config=HybridRetrieval(limit=40),
    )

def merge_tension_duplicates(hits):
    tension_hits = [m for m in hits if re.search(r"\d+\s*psi", m.content, re.I)]
    if len(tension_hits) < 2:
        return hits
    newest = max(tension_hits, key=lambda m: parse_ts(m.updated_at))
    # Canonical rewrite lets Engram transform reconcile; job removes stragglers after.
    run = client.memories.add(
        newest.content
        if "canonical" in newest.content.lower()
        else f"Canonical hoop tension on cooperage-hoop-drive-bench-4: {newest.content}",
        user_id=cooper,
        group="personalization",
        properties=bench,
    )
    status = client.runs.wait(run.run_id)
    if status.status != "completed":
        raise RuntimeError(f"merge failed: {status.status}")

    refreshed = inventory()
    survivors = []
    for m in refreshed:
        if not re.search(r"\d+\s*psi", m.content, re.I):
            survivors.append(m)
            continue
        if "canonical" in m.content.lower() or m.id == newest.id:
            survivors.append(m)
            continue
        archive.append({"id": m.id, "content": m.content})
        client.memories.delete(m.id, user_id=cooper, group="personalization")
    return inventory()

def prune_stale(hits):
    now = datetime.now(timezone.utc)
    kept = []
    for m in hits:
        age = now - parse_ts(m.updated_at)
        seasonal = "festival barrel run" in m.content.lower()
        if seasonal and age > STALE:
            archive.append({"id": m.id, "content": m.content, "reason": "stale"})
            client.memories.delete(m.id, user_id=cooper, group="personalization")
        else:
            kept.append(m)
    return kept

def nightly_maintenance():
    hits = inventory()
    before = len(hits)
    hits = merge_tension_duplicates(hits)
    hits = prune_stale(hits)
    return {
        "before": before,
        "after": len(hits),
        "archived": len(archive),
        "sample": [m.content for m in hits[:5]],
    }

print(nightly_maintenance())

The sketch is intentionally conservative. It merges only clear tension siblings and prunes only labeled seasonal notes past age. Widen rules after metrics prove you are not deleting useful rare facts.

How do you operate and measure maintenance jobs?

Emit counts for scanned scopes, merges attempted, deletes, failed runs, and remaining conflict pairs. Alert on delete spikes and on merge failure rates. Sample a few scopes manually each week through the Engram console Memories and Runs views so committed operations match your logs.

Separate job classes by risk. A frequent lightweight duplicate merge can run daily. Aggressive TTL deletes may run weekly with a dry-run mode first. Privacy wipe requests are not maintenance. They are urgent, targeted delete paths with their own audit trail.

Scheduled cleanup keeps the average scope healthy. Long-running agent fleets still need an explicit staleness detector for facts that look current, retrieve well, and are nevertheless wrong for today’s environment. That detector is the next operational concern.

Our next chapter, How do you detect staleness in long-running agent deployments?, focuses on finding memories that survived cleanup yet no longer match the world the agents act in.