How do you choose an embedding model for memory?

The previous chapter treated a switch between embedding models as something that eventually has to be managed carefully once it happens. This chapter steps back to the decision that comes before any of that: how does a team actually pick which embedding model to build a memory system on in the first place, given how many options exist and how differently they perform depending on what’s actually being asked of them?

Why Isn’t There Simply One Best Embedding Model to Reach For?

Embedding models vary along several dimensions at once: the languages and domains they were trained on, how many dimensions their output vectors carry, how much text they can process in a single pass, and how quickly they can actually produce an embedding at the volume a real system demands. A model that excels at general-purpose retrieval across everyday language might perform noticeably worse on specialized vocabulary it never saw much of during training, while a model built specifically for one narrow domain might underperform outside of it. There’s no single ranking that holds across every possible use case, only a best fit for a specific system’s actual data and actual constraints.

What Do Public Benchmarks Like MTEB Actually Tell a Team, and What Do They Leave Out?

The Massive Text Embedding Benchmark aggregates a model’s performance across many different underlying tasks, retrieval, classification, clustering, and more, giving a useful, standardized starting point for comparing a large field of candidates without having to test each one from scratch. But a model’s strong showing on that benchmark’s general-purpose datasets doesn’t guarantee it will perform equally well on a specific system’s own specialized content, since the benchmark’s training and evaluation data may bear little resemblance to whatever a particular application actually needs to search through. A model that leads the general leaderboard is not automatically the model that leads on the one task a specific system actually cares about most.

This gap is exactly why relying purely on a leaderboard ranking, without ever testing against real, representative data, risks choosing a model that looks strong on paper but underperforms in practice. A team is better served treating published benchmarks as an initial filter, narrowing a large field down to a manageable shortlist, rather than as a final verdict on which model to actually deploy.

How Should a Team Actually Test Whether a Shortlisted Model Fits Its Specific Content?

The most reliable evaluation comes from running a shortlisted model against a sample of a system’s own real queries and real stored content, then measuring how well it actually surfaces the results a person would judge as genuinely relevant. This kind of direct, task-specific test reveals things a general benchmark simply can’t, particularly whether a model correctly distinguishes between specialized terms that look superficially similar but mean something quite different within a specific domain’s own vocabulary. A model that performs beautifully on generic web text can still stumble badly on the exact kind of specialized language a particular system relies on most, and the only reliable way to know that in advance is testing directly against that system’s own representative content.

Does a Larger Embedding Model with More Dimensions Always Perform Better?

Larger models with higher-dimensional output vectors do tend to capture more nuance, but that improvement comes bundled with real costs: slower inference, larger storage requirements per stored vector, and a real risk that an oversized model actually overfits to its training data in ways that hurt its performance on the specific kind of content a system actually needs to handle. A genuinely large embedding dimension isn’t automatically better if a system’s actual queries and content don’t need that much nuance to be distinguished correctly, and the extra storage and latency cost of carrying larger vectors everywhere can outweigh a marginal accuracy gain that never actually shows up in practice.

Why Would a Domain-Specific Model Sometimes Beat a Much Larger General-Purpose One?

A model trained specifically on a narrow domain’s own vocabulary and typical phrasing can correctly distinguish between specialized terms that a general-purpose model, however large, was simply never exposed to closely enough during its own training to tell apart reliably. This means a smaller, domain-tuned model can genuinely outperform a much larger general-purpose competitor on exactly the kind of content that domain actually produces, even though the general-purpose model would likely win on a broad, generic benchmark covering many unrelated domains at once. Choosing based purely on overall model size, without accounting for whether a model actually understands a system’s specific domain, risks picking the wrong tool for the job in front of it.

How Does Weaviate Engram’s Flexible Model Support Let a Team Actually Apply This Kind of Domain-Aware Choice?

Weaviate Engram runs on top of Weaviate’s broad support for different embedding model providers, letting a team choose whichever model genuinely fits their specific domain rather than being locked into one fixed, general-purpose default. Consider a veterinary clinic chain’s patient-history assistant, where records span many different species and each species carries its own distinct clinical vocabulary that a purely general-purpose model might easily conflate:

from engram import EngramClient

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

client.memories.add(
    "This patient, an eight-year-old Quarter Horse gelding, presented with intermittent hind-limb lameness consistent with early-stage degenerative suspensory ligament desmitis, confirmed via ultrasound.",
    properties={"patient_id": "patient-equine-2291"},
)

A veterinarian searching this patient’s history later needs the embedding model to correctly recognize this exact specialized condition rather than confusing it with an unrelated, superficially similar-sounding term from a completely different species’ typical ailments:

results = client.memories.search(
    query="Any history of suspensory ligament issues in this horse?",
    properties={"patient_id": "patient-equine-2291"},
)

A general-purpose embedding model trained mostly on everyday, non-clinical language might reasonably struggle to distinguish this specific equine musculoskeletal condition from other, only vaguely related lameness terms, simply because it never encountered enough specialized veterinary text during training to draw that line precisely. A model chosen specifically because it performed well on this clinic chain’s own sample of real veterinary records, rather than one chosen purely for its general-purpose leaderboard ranking, is far more likely to represent this kind of specialized clinical language the way a working veterinarian actually needs it represented. Engram’s flexibility in which underlying embedding model powers a given deployment is exactly what lets a team make this kind of domain-aware choice deliberately, rather than accepting whatever general-purpose default happens to be easiest to reach for.

Choosing the right embedding model is ultimately about matching a model’s own strengths to the specific content and queries a system actually has to handle, evaluated directly rather than assumed from a general leaderboard. Once a model is chosen and memories are flowing in, a separate and equally practical concern emerges: how a person actually phrases a query in the first place, and how much that phrasing itself shapes whether a search finds what it’s actually looking for. Our next chapter, How should you formulate queries for memory recall?, takes up exactly that concern.