Short answer: BM25 is the standard keyword ranking score based on term frequency and how rare those terms are.
It turns inverted-index matches into a repeatable relevance number without embeddings. Production keyword search still leans on BM25 because it rewards exact terms while down-weighting common words.
Keyword search relies on comparing term frequency against overall rarity to rank its results, but “compare term frequency against rarity” is a description of an idea, not a precise, repeatable calculation. BM25 is the specific algorithm that turns that idea into an actual, well-defined score, and it remains the dominant keyword-ranking algorithm in production search systems today, entirely without needing an embedding model or a vector index anywhere in the picture.
What Problem Does BM25 Actually Improve on Compared to Simpler Term-Frequency Scoring?
A naive approach to scoring might simply count how many times a query term appears in a document and treat higher counts as automatically better. This breaks down quickly in practice: a document that mentions a term ten times isn’t necessarily ten times more relevant than one that mentions it once, and a term appearing rarely across a whole collection deserves more weight than a common word that shows up nearly everywhere. BM25 addresses exactly these two shortcomings, building on top of the earlier term-frequency, inverse-document-frequency approach by adding a smarter handling of both how much repeated occurrences should count, and how a document’s overall length should factor into its score.
Why Doesn’t Doubling the Number of Times a Term Appears Double the Score?
BM25 applies what’s called term-frequency saturation: as a term appears more and more times within the same document, each additional occurrence contributes progressively less to the overall score, rather than every additional mention counting equally. This reflects a genuine intuition about how relevance actually works: a document mentioning a term twice is probably meaningfully more relevant than one mentioning it once, but the difference between mentioning it fifty times versus fifty-one times says almost nothing extra about relevance. This saturating behavior is controlled by a tunable parameter, conventionally called k1, which determines how quickly that diminishing-returns effect kicks in.
Why Does a Document’s Length Need to Be Factored Into the Score at All?
Without any length adjustment, longer documents would have an unfair, mechanical advantage: they simply have more room to mention any given term multiple times, purely as a side effect of containing more text overall, not because they’re actually more relevant to it. BM25 corrects for this with a length-normalization component, conventionally controlled by a parameter called b, which reduces a document’s score somewhat if it’s considerably longer than the average document in the collection, and increases the relative weight of matches in shorter documents. Set this normalization to its maximum strength and length differences are fully accounted for, set it to zero and length stops mattering entirely, with most real systems landing somewhere between the two.
How Does BM25 Decide That a Rare Term Deserves More Weight Than a Common One?
This is the inverse-document-frequency half of the calculation: a term that appears in only a small fraction of documents across the entire collection contributes a much larger boost to the score of any document containing it, compared to a term that shows up in nearly every document and therefore carries almost no distinguishing power. A search for a specific technical term that appears in only a handful of stored memories out of thousands correctly treats a match on that rare term as a strong signal, while a search including a common word contributes comparatively little to the final ranking, exactly reflecting how little that common word actually tells you about which document is truly relevant.
Why Does BM25 Remain the Standard Choice Despite the Rise of Vector Search?
BM25 requires no training, no embedding model, and no vector index at all, it’s a purely statistical calculation over term counts that any system can compute directly and cheaply the moment content is indexed. This makes it fast, transparent, and immediately explainable: a person can see exactly which terms matched and roughly why a given document scored the way it did, something a vector similarity score doesn’t offer nearly as directly. It also performs reliably in exactly the situation vector search structurally struggles with, precise matching on specific, rare, or technical terms, which is precisely why BM25 remains the standard keyword half of hybrid search rather than being replaced outright by embeddings.
How Does Weaviate Engram Rely on BM25 Specifically Within Its Hybrid Retrieval?
Weaviate Engram’s hybrid retrieval option runs BM25 scoring alongside vector similarity, combining both into one ranked result set, giving a search the exact-term precision BM25 provides without sacrificing the semantic flexibility vector search contributes. Consider a stock-footage licensing library’s search assistant helping video editors find clips by exact camera or format terminology alongside broader descriptive searches, where BM25’s precision on rare, specific terms genuinely matters:
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
client.memories.add(
"Clip catalog note: this aerial coastline sequence was shot in ProRes 4444 at 5.9K on a gimbal-stabilized rig, licensed for broadcast use only, not available for social media licensing.",
properties={"clip_id": "clip-88214-coastline"},
)
A search for this exact clip benefits directly from BM25’s handling of a rare, specific technical term:
results = client.memories.search(
query="Do we have any ProRes 4444 aerial coastline footage available for broadcast?",
properties={"clip_id": "clip-88214-coastline"},
retrieval_config=HybridRetrieval(limit=5),
)
“ProRes 4444” is a rare, specific term that appears in only a small fraction of the library’s catalog entries, and BM25’s inverse-document-frequency weighting correctly treats a match on this specific format as a strong, distinguishing signal, exactly the kind of precision a purely semantic comparison alone might blur across similarly-worded but differently-formatted clips. Combined with vector search correctly understanding that “aerial coastline” and “coastline sequence” describe the same conceptual content even when worded slightly differently, this hybrid approach gives the editor exactly the precise, relevant match the search actually needed.
BM25 explains how keyword relevance gets scored on its own. Combining that keyword score with a vector similarity score into one single, coherent ranking requires its own deliberate mechanism, since the two scores aren’t naturally on the same scale or measuring the same thing. Our next chapter, What is hybrid search?, takes up exactly that combination.