Short answer: Store the durable facts pixels and audio reveal as text or pre-extracted items in Engram—not every JPEG—then hybrid-search those claims across sessions.
Multimodal agents see frames, hear clips, and read sensors. Text-only recall misses scorch marks and crack patterns once pixels leave the context window. Engram accepts text, conversations, and pre-extracted facts; your perception stack turns vision and audio into those shapes. Store captions and voice notes under user and property scopes (e.g. panel_id) with media URIs pointing at access-controlled blobs. Hybrid search recovers claims without replaying every frame. Prefer sensor evidence for visual tasks and spoken corrections when humans override false captions. Privacy follows the media—deleting only the image while leaving a detailed caption is not deletion. Keep raw media out of Engram; keep searchable judgments in.
Multimodal agents do not live in chat boxes alone. They look at frames, listen to clips, and read sensor overlays before they answer. A support bot that only remembers typed tickets will miss the scorch mark in a photo. A field agent that only stores transcripts will forget the crack pattern the camera already captured. Long-term memory for these systems has to survive the moment the pixels leave the context window.
That does not mean every JPEG must sit inside the memory service forever. It means the durable facts those pixels reveal must be written somewhere searchable. Weaviate Engram is built for that durable layer. It accepts text, conversations, and pre-extracted facts. Your perception stack turns vision and audio into those shapes. Engram then extracts, merges, and retrieves them across sessions.
Why does text-only recall fail once an agent can see and hear?
Text is a lossy projection of the world. A user says “the left hinge is loose.” The photo shows the right hinge and a missing screw. If memory only stores the utterance, the next turn inherits the error. Multimodal agents need a path from observation to durable claim. Research systems such as M3-Agent store faces, voices, and textual knowledge in an entity-centric graph for exactly this reason. Episodic clips capture what happened. Semantic nodes capture who and what remains true later.
Production stacks rarely ship a full multimodal graph on day one. They still need the same split. Keep short-lived media in object storage or a media index. Commit the lasting claims to Engram. Without that split, every new image competes with chat history for tokens, and last week’s visual finding disappears when the session resets.
So the design question is not “can Engram ingest raw pixels?” Engram’s public input types are string, conversation, and pre-extracted items. The design question is how your agent turns a multimodal moment into one of those inputs without throwing away the evidence pointer.
What should you store from an image or audio clip?
Store the decision-relevant abstraction, not the blob. For an image that means a short caption of the failure mode, measured values read from the frame, and an object key for the source file. For audio that means a transcript plus speaker and intent tags when they matter. Engram’s pipeline can extract structured memories from those strings. Transform steps can merge updates when a later photo revises an earlier claim.
Pre-extracted input is useful when your vision model already produced structured findings. You assign the topic yourself and skip Engram’s LLM extraction step. The content still flows through transform and commit. That keeps multimodal extractors in your control while Engram remains the merge and search layer.
Pointers matter. A memory that says “panel north rose shows copper green bloom near lead came” is useful. A memory that also records media_uri=s3://atelier/panel-north-rose/uv-2026-03-12.jpg lets the agent reopen the evidence when a human asks to see it. Engram holds the searchable claim. Object storage holds the bytes.
How do multimodal agents write and read Engram in practice?
Consider a conservation desk agent on stained-glass-atelier-7. A technician uploads a UV photo of a rose window panel and speaks a field note. A vision model describes the bloom. Speech-to-text captures the spoken note. The agent writes both into Engram as string observations scoped to the atelier user and the panel property. Later, when someone asks whether the copper green bloom returned after cleaning, hybrid search recovers the durable claim without replaying every frame.
import os
from engram import EngramClient, HybridRetrieval, PreExtractedInput, PreExtractedItem
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
atelier_user = "conservator-elena"
panel_scope = {"panel_id": "north-rose-lancet-3"}
# Perception stack already captioned the UV frame and transcribed the voice note.
vision_note = (
"UV frame uv-2026-03-12.jpg on stained-glass-atelier-7: copper green bloom "
"along the lead came at the lower left petal. Bloom covers roughly 4 cm. "
"media_uri=s3://atelier/panel-north-rose/uv-2026-03-12.jpg"
)
voice_note = (
"Technician: after solvent wipe the bloom lightened but a faint halo remains. "
"Do not apply heat near this came."
)
run_a = client.memories.add(
[vision_note, voice_note],
user_id=atelier_user,
group="continual_learning",
properties=panel_scope,
)
# Structured finding from the in-house vision tool, committed without re-extraction.
run_b = client.memories.add(
PreExtractedInput(items=[
PreExtractedItem(
content=(
"North rose lancet 3 retains a faint copper green halo after wipe; "
"heat treatments are forbidden on this came; "
"evidence=s3://atelier/panel-north-rose/uv-2026-03-12.jpg"
),
topic="UserKnowledge",
),
]),
user_id=atelier_user,
group="continual_learning",
properties=panel_scope,
)
hits = client.memories.search(
"Did the copper green bloom return on the north rose panel after cleaning?",
retrieval_config=HybridRetrieval(limit=6),
user_id=atelier_user,
group="continual_learning",
properties=panel_scope,
)
The agent answers from those hits. If the human wants the photo, the agent follows the stored media URI. Engram never had to become an image database. It became the memory of what the image meant for the job.
How should retrieval mix modalities without flooding the prompt?
Query with the user’s current text first. Hybrid retrieval in Engram blends vector similarity with keyword match, which helps when panel IDs and media filenames appear in the memories. If the turn includes a new image, caption it before search. Use that caption as the query, or concatenate it with the spoken request. Do not dump every prior frame into the prompt because a single photo arrived.
Separate lanes keep latency sane. A media index can return near-duplicate frames when visual similarity is the question. Engram returns the durable policy and history when the question is about prior findings. Many research stacks keep visual embeddings beside textual graphs for the same reason. Your product can keep those lanes distinct and still feel multimodal to the user.
Bridge carefully across sessions. A week later the technician may ask only in text. The captioned memories must still surface. That is the test of multimodal memory that actually works. If recall depends on re-uploading the same photo, you built a media cache, not an agent memory.
Where do teams usually go wrong with multimodal memory?
They store transcripts of captions and never store the decision. “Photo received” is not a memory. “Copper green bloom remains after wipe; heat forbidden” is. They also overwrite silently. A second UV shot that shows the bloom is gone should update the prior claim through Engram’s transform path, not leave two contradictory facts with equal weight and no timeline.
Another failure mode is modality privilege. Teams trust the typed chat more than the photo because chat is easier to log. Multimodal agents should treat conflicting channels as a conflict to resolve. Write both observations. Prefer the sensor evidence when the task is visual. Prefer the spoken correction when the human explicitly overrides a false automatic caption.
Privacy follows the media. Captions can still leak faces and locations. Scope Engram memories by user and panel. Keep raw media in access-controlled storage. Retention policy should cover both the blob and the claim. Deleting only the image while leaving a detailed caption is not deletion in practice.
Our next chapter, What are the security risks of prompt injection via stored memories?, turns from what multimodal agents remember to how hostile content can hide inside those memories and later steer retrieval.