Short answer: Keyword search finds documents by exact terms; an inverted index maps each term to the places it appears.
That older structure remains essential beside vectors. It makes exact-token lookup fast and fills the gap where semantic neighbors look related but do not contain the required identifier or phrase.
The previous chapter established that vector search alone structurally can’t guarantee exact matches on specific terms and identifiers. Keyword search is the older, complementary technique that fills exactly that gap, and understanding how it actually works, through a structure called an inverted index, explains why it remains essential even in a world where semantic vector search has become the more talked-about technology.
What Problem Does an Inverted Index Actually Solve?
Finding every document that contains a specific word by checking each document one at a time, reading through its full text looking for a match, is called a forward index approach, and it scales exactly as badly as brute-force vector comparison does: checking every document individually gets slower as the collection grows. An inverted index flips this relationship entirely. Instead of mapping documents to the words they contain, it maps words directly to the documents that contain them, so finding every document containing a specific term becomes a single, direct lookup rather than a scan through everything ever stored.
This is precisely the same principle behind the index at the back of a printed book: rather than reading the entire book start to finish to find every mention of a specific topic, a reader looks up the topic directly and gets pointed straight to the relevant pages. An inverted index does the identical thing for searchable text, just applied to entire documents rather than page numbers.
How Does Text Actually Get Turned Into Something an Inverted Index Can Use?
Before an inverted index can be built, text has to be broken down into individual searchable units called tokens, a process called tokenization. A sentence gets split into its constituent words, and those words become the entries the index actually maps. Once tokenized, the index builds what’s called a posting list for each distinct token: a record of every document containing that specific term, along with how many times it appears in each one. Searching for a specific word then means looking up its posting list directly, immediately retrieving exactly the documents containing that term, without touching any document that doesn’t contain it at all.
How Does an Inverted Index Actually Rank Its Results, Rather Than Just Finding Matches?
Simply knowing which documents contain a query term isn’t enough on its own, some matches are clearly more relevant than others, and a useful search needs to rank accordingly. Keyword search systems typically combine two signals to produce this ranking: how often a specific term appears within a given document, and how rare that term is across the entire collection overall. A term that appears frequently in one specific document, but rarely elsewhere in the broader collection, is treated as a strong, distinguishing signal that this document is genuinely about that specific term, rather than just mentioning it in passing the way a common, frequently occurring word might.
This combination of signals, term frequency weighed against overall rarity, is what lets keyword search produce a genuinely useful ranked list rather than just an unordered pile of documents that happen to contain a matching word somewhere.
Why Does Keyword Search Remain Genuinely Useful Even Though Vector Search Understands Meaning?
Keyword search offers something vector search structurally can’t guarantee: predictable, exact matching on precise terms, exactly the gap identified in the previous chapter. A search for a specific alphanumeric identifier, a rare technical term, or an exact proper noun benefits enormously from a mechanism built specifically to find precise character matches rather than conceptually similar ones. This predictability is also valuable in its own right beyond just precision: a keyword match is easy to explain, since a person can see exactly why a specific document matched, the term they searched for is literally present in it, something a vector similarity score alone doesn’t offer nearly as transparently.
How Does Weaviate Engram Rely on This Inverted-Index Machinery Alongside Vector Search?
Weaviate Engram’s hybrid retrieval option combines vector similarity with exactly this kind of keyword-based, inverted-index-powered matching in a single search call, so a query benefits from both semantic flexibility and precise, exact-term matching at once. Consider a museum collections registrar’s assistant helping staff track artifacts by their exact accession numbers alongside broader descriptive context, where both kinds of matching genuinely matter in the same search:
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
client.memories.add(
"Accession 1987.44.12, a bronze ceremonial vessel from the museum's East Asian collection, was reexamined this year and its provenance updated to reflect a confirmed acquisition date of 1987, not the previously assumed 1985.",
properties={"collection_area": "east-asian"},
)
A registrar searching for this exact item benefits from both search mechanisms working together in one call:
results = client.memories.search(
query="What do we know about accession 1987.44.12's provenance?",
properties={"collection_area": "east-asian"},
retrieval_config=HybridRetrieval(limit=5),
)
The exact accession number in this query benefits directly from the inverted-index-backed keyword matching this chapter has described, since “1987.44.12” needs to match precisely rather than approximately, exactly the kind of identifier vector similarity alone can’t reliably guarantee. The surrounding language about provenance and reexamination benefits from vector search’s semantic flexibility, correctly connecting related concepts even if phrased somewhat differently from how the original memory was worded. Neither mechanism alone would serve this registrar as well as the two working together, which is exactly why Engram’s hybrid retrieval, and the underlying inverted index quietly powering its keyword half, matters as much as the vector search this Part has spent so much time explaining.
Keyword search relies on counting term occurrences and comparing them against overall rarity to rank results. The specific algorithm most modern keyword search systems actually use to turn those two signals into a precise relevance score deserves its own closer look. Our next chapter, What is BM25?, takes up exactly that algorithm.