Short answer: It compresses embedding numbers to fewer bits so vectors take less space and search faster, with some precision loss.
Full-precision floats get expensive at scale. Quantization shrinks each dimension deliberately. Systems often search with compressed vectors first, then rescore a short candidate list at full precision so storage savings apply broadly while final ranking stays accurate. Different schemes trade aggressiveness for accuracy; Engram-scale memory depends on that tunable infrastructure.
Multi-vector representations trade additional storage for finer-grained precision, but that isn’t the only place storage cost shows up in vector search. Even ordinary, single-vector embeddings accumulate real memory cost at scale, and vector quantization is the family of techniques built specifically to shrink that cost, deliberately trading away some precision in exchange for meaningfully smaller, faster-to-search vectors.
Why Does Storing Vectors at Full Precision Become a Genuine Cost at Scale?
An ordinary vector embedding typically stores each of its numbers as a 32-bit floating-point value, and a vector with hundreds or thousands of dimensions adds up quickly: a single 768-dimension vector at full precision already takes over three thousand bytes, and that cost multiplies directly by however many memories a store has accumulated. For a handful of stored items this is negligible. For a memory store that’s grown to millions of entries over years of continuous use, the raw storage and, more importantly, the memory needed to keep an entire searchable index loaded and fast, becomes a real, measurable cost that grows in direct proportion to how much has been stored.
What Does Quantization Actually Do to a Vector to Shrink It?
Quantization reduces the precision used to represent each number in a vector, the same basic idea as rounding a number to fewer decimal places, applied systematically across an entire vector. Instead of storing each dimension as a full 32-bit float, a quantized representation might store it as an 8-bit integer, a single bit, or some other compressed form, dramatically reducing the total space each vector occupies. This is a genuinely lossy operation: information is discarded in the process, and a quantized vector is necessarily a coarser, less precise approximation of the original than the full-precision version it replaced.
A useful way to picture this tradeoff is thinking of a full-precision vector as a complete street address, precise enough to locate one specific house exactly, and a quantized version as something coarser, perhaps just the city. The coarser version takes far less space to store, but it can no longer distinguish between two different houses that happen to be in that same city, exactly the kind of precision loss quantization deliberately accepts in exchange for its space savings.
Does Losing This Precision Actually Break Search, or Just Make It Slightly Less Accurate?
It doesn’t break search outright, but it does measurably affect recall, the fraction of genuinely relevant results a search actually manages to find. Two vectors that were meaningfully different before quantization can end up looking identical afterward if they both happen to round into the same coarser, compressed representation, meaning a search can no longer reliably tell them apart even though they represented genuinely distinct content beforehand. How much this actually matters depends heavily on how aggressively a system chooses to compress, since lighter compression preserves nearly all of the original precision, while more aggressive compression trades away considerably more accuracy in exchange for correspondingly larger space savings.
How Do Systems Recover Most of the Lost Accuracy Without Giving Up the Storage Savings Entirely?
A common and effective pattern keeps both versions available: the compressed vectors get used for the fast, initial pass across an entire large collection, quickly narrowing things down to a modest set of promising candidates, and then the original, full-precision versions of just that narrowed-down set get pulled in for one final, accurate rescoring pass. This gets the best of both situations: the memory savings of compression apply across the full collection, where the volume actually matters, while the final ranking that a user or application actually sees benefits from full, uncompressed accuracy, since that final rescoring only has to run against a small handful of candidates rather than the entire store.
Are All Quantization Approaches Equally Aggressive, or Do They Offer Genuinely Different Tradeoffs?
Different quantization techniques land at meaningfully different points along the same underlying tradeoff, some compressing more conservatively and preserving nearly all of the original accuracy, others compressing far more aggressively and sacrificing noticeably more precision in exchange for correspondingly larger space savings. None of these approaches is universally correct, the right choice depends on how much a specific system can actually tolerate in recall loss weighed against how much storage and cost pressure it’s actually under, which is exactly why this remains a deliberate configuration decision rather than something with one single, always-correct answer.
Why Does This Underlying Infrastructure Matter for a Memory System Like Engram, Even Though It Operates Beneath the Level a User Ever Interacts With Directly?
Weaviate Engram’s memory search runs on top of Weaviate’s underlying vector storage, meaning the same compression techniques covered in this chapter are exactly what let a memory store keep growing for years without the cost of keeping it searchable growing at an unsustainable rate. Consider a citizen-science nature-observation platform where volunteers across an entire country log species sightings continuously, accumulating millions of individual observation records over the platform’s lifetime:
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
client.memories.add(
"Observation logged near the eastern ridge trail: a pair of peregrine falcons nesting on the cliff face, first confirmed breeding pair reported in this specific location in over a decade.",
properties={"region_id": "ridge-conservation-area"},
)
A volunteer searching this ever-growing archive years later still expects a fast, accurate response, and the underlying compression covered in this chapter is exactly what makes that possible at the platform’s actual, accumulated scale:
results = client.memories.search(
query="Have peregrine falcons nested in this area before?",
properties={"region_id": "ridge-conservation-area"},
retrieval_config=HybridRetrieval(limit=5),
)
Without compression, keeping millions of accumulated observation vectors searchable in memory would become an increasingly expensive, eventually unsustainable cost as the platform’s volunteer community kept logging more sightings year after year. With it, the platform can keep growing its archive indefinitely without that growth translating into an unmanageable operating cost, precisely because the underlying infrastructure Engram is built on top of was designed from the outset to make exactly this tradeoff available and tunable, rather than forcing every deployment to accept full, uncompressed storage cost regardless of scale.
Quantization addresses the raw storage cost of vectors themselves. A separate, closely related question is how a search combines this kind of similarity matching with ordinary, structured filtering, narrowing results down by specific property values alongside whatever similarity search is already doing. Our next chapter, What is filtered vector search?, takes up exactly that combination.