What is multi-modal memory?

Everything discussed in this Part so far has quietly assumed memory is made of text, a fact, a preference, a summary, all expressible as sentences. Plenty of what an agent might genuinely need to remember doesn’t start out as text at all. A photo, an audio clip, a video, a diagram, all carry information a text-only memory system has no native way to capture, and pretending otherwise means losing exactly the detail that made the original content worth remembering in the first place.

Why Can’t Non-Text Content Just Be Converted to a Text Description and Stored Normally?

It often can, and for a lot of practical purposes this is a perfectly reasonable approach: describe an image in words, and store that description as an ordinary text memory. The limitation is that any description is a lossy compression of the original, capturing only whatever the person or system writing it happened to think was worth mentioning. Two people describing the same photo will emphasize different details, and neither description fully recreates what’s visually present in the source.

This matters more in some domains than others. A photo of a whiteboard sketch might be perfectly well served by a text description of its key points. A photo documenting a specific physical detail, exact positioning, a subtle visual defect, a precise gesture, loses something meaningful the moment it’s reduced to a sentence, because the sentence can only capture what its author consciously noticed and chose to write down.

What Does It Actually Mean to Search Across Modalities Rather Than Just Within Text?

True multi-modal memory means being able to retrieve non-text content based on its own characteristics, not just a text description standing in for it, and ideally being able to search across modalities so a text query can surface a relevant image, or an image can surface relevant text. This is possible because certain embedding models can represent multiple modalities, images, audio, video, and text, in the same shared vector space, meaning conceptual similarity can be measured across a modality boundary rather than only within one.

This unified space is what makes something like “find the photo that best matches this description” possible in the first place, since the description’s vector and the image’s vector both live in a space where genuine conceptual closeness is measurable, rather than requiring the image to first be reduced entirely into words before any comparison can happen.

What Actually Goes Wrong When a System Treats Everything as Text-Only From the Start?

Forcing every non-text input through a text-only pipeline means whatever gets discarded in the initial conversion, is gone permanently and can never be recovered by a smarter or more careful retrieval step later. If a photo only ever gets reduced to “user shared a picture of a shoulder exercise,” any later question depending on visual detail the description happened to leave out, exact arm angle, whether a particular joint looked correctly aligned, simply has nothing to draw on, because that detail was never captured anywhere the system could later retrieve it from.

This is the same underlying lesson already covered when discussing lossy compression elsewhere in memory design: information thrown away at write time can’t be recovered at read time, no matter how good the retrieval mechanism eventually becomes. Getting the initial capture right matters more than trying to compensate for a lossy capture with a clever search step downstream.

Does Every Application Actually Need True Multi-Modal Retrieval, or Is a Text Description Often Enough?

Plenty of applications are genuinely fine with the text-description approach, and reaching for full multi-modal infrastructure when a simple description would serve just as well adds real complexity without a matching benefit. The deciding question is whether the fine visual, auditory, or spatial detail in the original content is something a later retrieval genuinely needs to reconstruct, or whether a competent summary captures everything that’s actually going to matter. If nobody will ever need more than “there was an image of X,” a description suffices. If the specific visual or auditory characteristics themselves are the thing worth recalling, a description alone will eventually fall short.

How Does Weaviate Engram’s Foundation Support Extending Memory Beyond Text?

Weaviate Engram’s memory pipeline today works with text-based input, string content, conversations, and pre-extracted facts, but it’s built directly on top of Weaviate, which natively supports multi-modal vector search through vectorizer integrations that embed images, audio, and video into the same space as text. This means a system combining Engram with Weaviate’s own collections can capture the best of both: durable, reconciled memory for what can be expressed as facts, alongside true multi-modal vectors for content where the visual or auditory detail itself matters. Consider a physical-therapy practice reviewing patient exercise-form videos, where a clinician’s textual observations and the actual video clips both need to be retrievable:

from engram import EngramClient

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

client.memories.add(
    "Patient's shoulder external rotation still shows slight compensation from the trunk; cleared to progress resistance band tension next session.",
    user_id="patient-6602",
)

This textual clinical observation is exactly the kind of durable, reconciled fact Engram’s pipeline handles well, updated over successive sessions as the patient’s form improves. The actual video clip of that session’s exercise, where the specific angle and quality of movement matters far more than any single sentence could capture, is stored separately in a Weaviate collection configured with a multi-modal vectorizer, letting a clinician later search across visually similar movement patterns directly, or even retrieve relevant clips using a text query describing the movement pattern they’re looking for. Neither piece replaces the other. The textual memory gives durable, reconciled clinical continuity across sessions, while the video’s own vector representation preserves the specific visual detail that a sentence, however carefully written, would have had to leave behind.

Multi-modal memory extends what kind of raw content a memory system can meaningfully capture. A separate, higher-level question remains regardless of modality: whether relying on any single memory category covered so far, working, episodic, semantic, procedural, or reflective, is ever actually sufficient on its own for a real agent. Our next chapter, Why is one memory type never enough?, closes out this Part by taking up exactly that question.