What is memory in human-in-the-loop agent systems?

Human-in-the-loop systems pause agents for approvals, overrides, and corrections. Memory is what keeps those pauses honest. Without durable memory, an agent forgets why it stopped, invents a softer constraint after the human leaves, or treats a rejected action as still open. With durable memory, approvals become facts, rejections become constraints, and human feedback becomes searchable experience. This chapter explains what HITL memory must record, how to separate pending state from ratified decisions, how Weaviate Engram stores human corrections and case-scoped approvals, and how agents should resume only after those memories are committed.

Why Does Human-in-the-Loop Fail Without Durable Memory?

A pause that lives only in process RAM is not a pause. It is a hope. Servers restart. Workers move. Reviewers reply hours later. If the only record of the pending action sat in a local variable, the resumed agent rebuilds a story from scraps. It may re-propose the same risky tool call. It may skip a condition the reviewer already stated.

HITL also fails when human words stay trapped in a ticket comment that the agent never searches. The reviewer writes that heritage oaks need arborist sign-off before removal. The next agent run never sees that sentence. Memory must capture the human decision in the same store the agent uses for facts.

The goal is continuity across people and machines. The agent proposes. A human decides. Engram remembers. The next agent turn reads the decision as first-class context, not as optional folklore.

What Should Be Remembered at Each Approval Gate?

Once the failure mode is clear, selection matters. Not every keystroke from a reviewer belongs in long-term memory. Store the decision, the binding constraints, and the case identity. Leave UI chatter and draft notes out unless they change future behavior.

Three memory shapes cover most gates. A pending proposal records what the agent wanted to do and why it stopped. An approval or rejection records the human outcome in plain language. A correction records how future similar cases should behave. Pending proposals can be case-scoped and short-lived. Approvals should be durable for audit. Corrections may graduate into project-wide experience when the lesson is safe to share.

Keep provenance in the text. Name the reviewer role and the timestamp in the memory content when your application needs audit trails. Engram will extract and organize what your topics allow. Clear source language makes later searches trustworthy.

How Does Weaviate Engram Hold HITL Decisions Beside Agent Facts?

The shapes still need a store. Weaviate Engram fits because humans and agents can write into the same scoped case. An intake agent adds observations with a case id property. When a sensitive tool needs approval, your application writes a pending proposal memory and waits. After the reviewer acts, write the outcome with memories.add, then runs.wait before resuming the agent. The resume search should require those approval memories, not only the original observations.

Use separate groups when needed. Case operations can live in one group. Project-wide lessons from trusted reviewers can live in a continual learning group with project-wide topics. Untrusted end-user feedback should stay user-scoped so one person cannot rewrite behavior for everyone. Engram’s scoping model makes that policy enforceable rather than aspirational.

Here is a municipal tree-removal case where an agent pauses for arborist approval and stores the human decision in Engram:

import os
from engram import EngramClient, HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
case_id = "tree-case-4417"
group = "permit_hitl"

# Agent proposes a sensitive action and records the pending gate
pending = client.memories.add(
    "Agent proposes issuing provisional removal for the oak at 18 Cedar Lane. "
    "Reason: trunk lean toward sidewalk. Waiting for arborist approval before any cut order.",
    user_id="inspector-lee",
    group=group,
    properties={"case_id": case_id, "gate": "arborist"},
)
client.runs.wait(pending.run_id)

# Human reviewer decides later; application writes the outcome
decision = client.memories.add(
    "Arborist M. Okonkwo rejects provisional removal for tree-case-4417. "
    "Require a cable brace assessment first. "
    "Do not generate a cut order until brace assessment is filed.",
    user_id="inspector-lee",
    group=group,
    properties={"case_id": case_id, "gate": "arborist", "decision": "reject"},
)
client.runs.wait(decision.run_id)

# Trusted procedural lesson for future similar cases
lesson = client.memories.add(
    "For sidewalk-lean heritage oaks, always require arborist brace assessment before removal proposals.",
    group="permit_lessons",
)
client.runs.wait(lesson.run_id)

# Agent resumes: search case decisions, then institutional lessons
case_mem = client.memories.search(
    query="arborist approval removal cut order brace assessment",
    user_id="inspector-lee",
    group=group,
    properties={"case_id": case_id},
    retrieval_config=HybridRetrieval(limit=6),
)
lessons = client.memories.search(
    query="heritage oak sidewalk lean brace assessment before removal",
    group="permit_lessons",
    retrieval_config=HybridRetrieval(limit=4),
)

The resumed agent sees a rejection as memory, not as a missing button click. The cut order tool stays blocked because the constraint is now searchable and durable.

How Should Agents Resume Without Replaying the Whole Review?

After decisions are stored, resume prompts still need discipline. Load the latest gate outcome for the case first. Load only the observations needed for the next allowed step. Do not paste the entire reviewer chat. HITL memory works when the agent treats the decision as authoritative and moves to the next legal action.

If search returns no decision for an open gate, do not invent approval. Re-enter the waiting state. Empty results after a known human action usually mean the write has not committed yet or the wrong case id was used. Wait on the decision run id when your orchestrator still holds it. Otherwise retry the scoped search briefly before escalating to a human again.

Also separate sticky preferences from one-case rulings. A reviewer who always rejects unverified auto-emails can contribute a user-scoped preference. A ruling about one oak stays on that case id. Mixing those scopes is how one dramatic case rewrites an entire department’s behavior by accident.

What Governance Rules Keep Human Memory Trustworthy?

Resume hygiene is not enough without write governance. Only authorized roles should write approval and rejection memories. End users can supply evidence. They should not silently mint institutional lessons. Route trusted reviewer outcomes into the lessons group. Keep citizen comments in a case-scoped evidence topic.

Prefer pre-extracted facts when the UI already structured the decision as approve or reject. That reduces extraction ambiguity for high-stakes gates. Use free-text strings for richer reviewer rationale. Both paths can coexist in Engram. The important part is waiting for commit before the agent continues.

Over time, HITL memory becomes the audit trail your operators already wished they had. Agents stay fast on routine steps. Humans stay authoritative on irreversible ones. Engram keeps both sides speaking the same case language after every pause.

Our next chapter, What is federated memory across independent agent deployments?, leaves the single-project HITL desk and asks how memory can cooperate when agents live in separate deployments that cannot share one naive store.