How do you human-evaluate agent personalization quality?

A/B hit rates tell you whether the right string appeared in retrieval. They do not tell you whether personalization felt respectful, relevant, or correctly quiet. Humans still judge whether an agent used memory the way a good colleague would. This chapter frames human evaluation for memory-backed personalization, which rubrics separate recall from over-sharing, how Weaviate Engram search packs evidence for raters, and how to keep annotation trustworthy without turning every release into a focus group.

Why Can’t Automated Memory Metrics Finish the Job Alone?

Personalization quality is partly objective and partly social. Retrieving “prefers linen thread on folio nine” is objective. Deciding whether to mention that preference on a shipping-status question is social. Automated probes catch absence and contradiction. They struggle with taste, tone, and deliberate silence.

Benchmarks for personalized agents increasingly split criteria into memory presence and forgetting absence. Humans remain the calibration layer. Studies that validate LLM judges against people still report the need for agreement checks. Inter-annotator kappa in the high eighties is a target, not a guarantee you can skip humans forever.

Engram’s Personalization template and user-scoped topics like UserKnowledge make the memory side inspectable. Human eval asks whether the agent used those memories wisely in the reply the user actually saw.

What Should a Personalization Rubric Ask Raters to Score?

After you accept that humans are required, give them atomic questions. Prefer binary criteria over vague five-point vibes. Did the reply honor the stated binding preference? Did it avoid citing a preference the user revoked? Did it stay silent on irrelevant personal facts? Did it invent a preference that memory never supported?

Separate remembering, reasoning, and recommending when your product spans those modes. A status update needs light personalization. A materials recommendation needs denser use of stored taste. Scoring both with one “felt personal” slider hides failures. Preference score frameworks from workflow benchmarks keep participant-specific rubrics and score each applicable criterion independently for the same reason.

Always show raters the memory evidence and the final reply. Blind them to model version when comparing systems. Ask whether a reasonable client would expect that fact to be remembered, and whether the phrasing overstates certainty. Over-confident summaries are a common human-caught bug. “Hates cloth cases” may be wrong if the client only refused cloth for one humid shipment.

How Do You Build an Evaluation Packet From Weaviate Engram?

Raters need a packet, not a raw database dump. For each sampled session, include the user question, the agent reply, and the Engram hits that were injected into the prompt. Add the durable preferences you believe should apply. Add any revoked facts that must stay absent. That packet lets humans score presence and forgetting without hunting through logs.

Pull memories with the same scopes the agent used. Filter to personalization topics when the question is about taste. Keep job facts in a separate topic slice when the question is about process. Mixing everything into one undifferentiated list trains raters to reward chatter.

Here is a bindery desk helper that assembles a human-eval packet for one folio client after searching Engram:

import os
import json
from engram import EngramClient, HybridRetrieval

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
binder = "binder-opal"
client_user = "client-rowan-folio"
group = "bindery_desk"
folio = "folio-bind-9"

question = "Can you suggest endpapers for the poetry folio shipping next week?"
# In production, assistant_reply comes from your agent turn log
assistant_reply = (
    "For folio-bind-9 I would stay with soft Hahnemuhle endpapers in warm ivory, "
    "matching your earlier note that you prefer matte fiber over glossy stock."
)

memories = client.memories.search(
    query=question,
    user_id=client_user,
    group=group,
    topics=["UserKnowledge"],
    properties={"folio_id": folio},
    retrieval_config=HybridRetrieval(limit=6),
)

packet = {
    "case_id": "bindery-eval-204",
    "folio_id": folio,
    "question": question,
    "assistant_reply": assistant_reply,
    "retrieved_memories": [
        {"id": m.id, "topic": getattr(m, "topic", None), "content": m.content}
        for m in memories
    ],
    "rubric": [
        {
            "id": "pref_presence",
            "text": "Reply uses the client's stated endpaper or finish preference when recommending materials.",
            "expected": "yes",
        },
        {
            "id": "no_revoked",
            "text": "Reply avoids revoked or superseded finish preferences.",
            "expected": "yes",
        },
        {
            "id": "no_irrelevant_personal",
            "text": "Reply omits unrelated personal details that do not help the materials choice.",
            "expected": "yes",
        },
        {
            "id": "no_invented_pref",
            "text": "Reply does not invent a preference absent from retrieved memories and recent chat.",
            "expected": "yes",
        },
    ],
}

print(json.dumps(packet, indent=2))
# Hand packet to two raters; resolve disagreements before promoting a memory config.

The search mirrors production. The rubric mirrors what personalization should do. Humans mark each criterion. Disagreements become rubric fixes, not silent averages.

How Do You Keep Human Eval Reliable and Affordable?

Sample stratified sessions. Include preference-heavy asks, status asks that should stay quiet, and correction turns after a preference change. Two raters per item with adjudication beats one rater at volume. Track agreement. If kappa collapses, the rubric is ambiguous.

Use humans to calibrate automated judges, then let judges draft at scale. Re-check a slice with people every release. Memora-style work shows strong but imperfect agreement between judges and humans. That is enough to accelerate screening. It is not enough to retire annotation on high-stakes personalization.

Protect privacy in packets. Redact secrets. Scope by user_id so raters never see another client’s memories. Engram’s user isolation helps here when your eval harness passes the correct id every time.

When Should Human Scores Block a Ship?

Gate releases on the criteria that hurt trust most. Invented preferences and revoked-preference reuse are ship blockers. Mild under-personalization on a shipping FAQ may be a backlog item. Pair human gates with the regression suite in the next chapter so pipeline edits cannot silently undo a hard-won rubric win.

Publish a short scorecard per release: presence rate, forgetting rate, over-share rate, and rater agreement. Tie dips back to Engram topic descriptions, retrieval limits, or prompt instructions. Personalization is a system property, not a model vibe.

Human evaluation is how memory-backed agents earn the word personal. Build atomic rubrics. Pack Engram evidence beside the reply. Sample for silence and forgetting, not only for recall. Then let people decide whether the agent remembered like a professional.

Our next chapter, How do you regression-test memory pipeline changes?, shows how to freeze that quality bar so a pipeline tweak cannot quietly undo what human raters already approved.