Short answer: They store many vectors for one item, one per meaningful piece, instead of a single pooled embedding.
A normal embedding collapses a sentence, document, or image into one point and can blur internal detail. Multi-vector form keeps localized meaning so late interaction and similar methods can match finer parts. The cost is more storage and comparison work, so it pays off most for multi-topic or bundled content, not every simple single-fact memory.
Late interaction relied on a specific idea that deserves examining directly rather than only through that one application: representing a single piece of content with many vectors instead of one. This idea, called multi-vector representation, turns out to be broader than just token-level text matching, applying just as naturally to images, documents, and other content where a single, pooled vector risks flattening away distinctions that actually matter.
What Exactly Distinguishes a Multi-Vector Representation From an Ordinary Single Vector?
An ordinary embedding represents an entire object, a sentence, a paragraph, an image, as one fixed-length list of numbers, a single point sitting somewhere in the meaning-space this Part has spent so much time describing. A multi-vector representation instead represents that same object as a whole set of vectors, one for each smaller, meaningful piece the object can be broken into, a token in a sentence, or a small patch of an image. Rather than collapsing all of those pieces down into one averaged point, a multi-vector representation keeps each piece’s own distinct position in the space, preserving exactly the kind of internal structure a single pooled vector would necessarily lose.
Why Does Breaking Content Into Many Smaller Pieces Actually Improve How Well It Gets Represented?
Any single object usually contains several genuinely distinct ideas or regions, not one uniform, undifferentiated meaning. A product review might discuss both price and build quality, two fairly separate concerns bundled into one passage. A photograph might contain both a foreground subject and a distinctive background detail, two visually distinct regions bundled into one image. Pooling everything down into a single vector forces all of these distinct pieces to blend together into one averaged representation, which works reasonably well when a query is broadly about the whole object, but works poorly when a query is specifically about just one of those bundled-together pieces. Keeping a separate vector per meaningful piece lets a query about any one specific piece find and match that piece directly, rather than having its relevance diluted by everything else bundled alongside it.
Does This Idea Only Apply to Breaking Up Text Into Individual Words?
It extends well beyond text. An image can be broken into a grid of smaller patches, each embedded on its own, letting a search query match against one specific, localized detail within an image rather than the image’s overall gist. A long document can be represented not just token by token, but with each individual page kept as its own distinct representation, letting a search correctly identify which specific page of a lengthy document actually contains the relevant material, rather than treating an entire multi-page document as one undifferentiated whole. The underlying principle, preserving distinct, localized meaning rather than collapsing it into one average, applies identically regardless of what kind of content is actually being represented.
What Does It Actually Cost to Represent Content This Way Instead of With One Vector?
Storing many vectors per object instead of just one requires meaningfully more storage overall, since the total footprint now scales with how many meaningful pieces an object contains rather than just with how many objects exist. It also requires more computation at comparison time, since a query now has to be compared against every one of those pieces rather than against just one summary vector. This cost has been substantially reduced through compression techniques that shrink each individual piece’s vector down while preserving most of its useful distinguishing detail, but it remains a genuine tradeoff, one worth making deliberately for content where this kind of fine-grained, localized precision earns its cost, rather than defaulting to it everywhere regardless of whether a simpler, single-vector representation would have served just as well.
When Does This Kind of Fine-Grained Representation Actually Earn Its Extra Cost for Memory Specifically?
Multi-vector representation earns its cost when a single stored memory genuinely bundles together several distinct sub-topics or details that a future query might need to match individually, rather than as an undifferentiated whole. Consider a wildlife-conservation field-survey assistant helping researchers log detailed observation notes from a single site visit, where one note might genuinely bundle together several distinct, independently-relevant observations:
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
client.memories.add(
"Site visit combined three distinct findings: a confirmed sighting of the endangered marsh warbler near the northern reed bed, evidence of invasive knotweed spreading along the eastern trail, and a water-quality reading showing elevated nitrate levels near the outflow.",
properties={"site_id": "wetland-survey-14"},
)
A single pooled embedding of this entire note risks blending three genuinely distinct findings, an endangered-species sighting, an invasive-species concern, and a water-quality reading, into one averaged representation that doesn’t strongly match a specific query about any one of them individually:
results = client.memories.search(
query="Have we recorded any recent knotweed sightings at this wetland site?",
properties={"site_id": "wetland-survey-14"},
retrieval_config=HybridRetrieval(limit=5),
)
A single-vector representation of the full, combined note might still surface this result reasonably well thanks to hybrid search’s keyword component correctly matching “knotweed” directly, but a multi-vector representation capable of matching the query specifically against just the invasive-species portion of the note, rather than the note’s blended average, would represent this kind of bundled, multi-topic content more faithfully still. This is exactly the kind of tradeoff this chapter has been describing: genuinely bundled, multi-topic content benefits from preserving its internal structure rather than collapsing it, while simpler, single-topic memories are served perfectly well by the ordinary single-vector approach covered throughout the rest of this Part.
Multi-vector representation trades additional storage and computation for finer-grained precision. That tradeoff, and the compression techniques that make it more affordable, deserve their own direct, dedicated treatment covering how vectors get shrunk without losing too much of what makes them useful in the first place. Our next chapter, What is vector quantization?, takes up exactly that subject.