What is reranking?

Alpha explained how a hybrid search balances vector and keyword signals into one initial ranked list. That list is still produced by models built for speed at scale, and speed at scale comes with a real accuracy cost. Reranking is the deliberate second pass that trades some of that speed back for a meaningfully more careful judgment, applied only where it actually matters, to the handful of candidates an initial search already narrowed things down to.

Why Can’t the Fast, Initial Search Already Be as Accurate as Possible From the Start?

The embedding models covered earlier in this Part, sometimes called bi-encoders, work by converting a query and every stored item into vectors independently, ahead of time, so that comparing them later is just a fast distance calculation. This speed is exactly what makes searching millions of items in milliseconds possible, but it comes at a cost: the query and each stored item never actually get to interact with each other during that comparison, each was embedded on its own, in isolation, with no awareness of what it would eventually be compared against.

A different kind of model, a cross-encoder, takes a query and one specific candidate together as a single joint input, letting the model directly consider how the two relate to each other in that specific pairing, producing a noticeably more accurate relevance judgment as a result. The catch is that this joint comparison can’t be precomputed the way independent embeddings can, it has to be freshly calculated for every single query-candidate pair, which makes it far too slow to run against an entire collection the way the initial fast search does.

How Does Reranking Actually Combine the Speed of One Approach With the Accuracy of the Other?

Reranking runs the fast, initial search first, exactly as covered throughout this Part, to quickly narrow a massive collection down to a small handful of promising candidates. Only then does the slower, more accurate cross-encoder step in, running its more careful, pairwise comparison against just that small set rather than against the entire original collection. This two-stage structure is often compared to a fisherman first casting a wide net to catch a large haul likely to contain what they’re after, then carefully sorting through that smaller catch by hand to pick out exactly the right fish. Neither stage alone would work as well: the wide net alone catches too much irrelevant material, and the careful hand-sorting alone would take forever applied to the entire ocean rather than one modest catch.

Does Reranking Retrieve New Candidates, or Only Reorder Ones Already Found?

Reranking doesn’t go looking for anything the initial search missed, it strictly reorders the exact same set of candidates the first search already retrieved, based on a more careful judgment of how relevant each one actually is. This means reranking can only improve the ordering of what’s already present, it can never rescue a genuinely relevant item that the initial search failed to surface in the first place. This is exactly why the size of that initial candidate set matters: if the first-stage search is set to retrieve too few candidates, reranking has nothing meaningful left to work with, no matter how good the reranking model itself might be.

When Does the Extra Cost and Latency of Reranking Actually Justify Itself?

Reranking adds genuine cost, both in computation and in response time, since it’s deliberately running a slower, more expensive model as an extra step after the fast search has already finished. This tradeoff makes the most sense specifically when getting the very top results exactly right matters a great deal, and when the underlying content is subtle enough that a fast, independent-embedding comparison genuinely struggles to distinguish a close match from the truly correct one. It matters less for situations where any one of several roughly-similar results would serve equally well, since the added cost buys precision that isn’t actually needed there.

How Does Weaviate Engram’s Retrieval Pipeline Accommodate a Reranking Step for Memory Search?

Weaviate Engram’s retrieval configuration returns a ranked, scored candidate set exactly suited to being handed off to a reranking step when a specific search genuinely needs that extra precision, letting an application add this second pass selectively rather than paying its cost on every single query. Consider a rare-book dealer’s cataloging assistant helping a specialist find the exact edition a demanding collector actually wants among many superficially similar catalog entries:

from engram import EngramClient

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

client.memories.add(
    "First edition, first printing of this title identified by the unclipped dust jacket and the uncorrected 'freind' typo on page 47, distinct from the more common second-state printing lacking that error.",
    properties={"title_id": "catalog-item-8834"},
)

An initial hybrid search across the full catalog quickly narrows down a modest set of candidates worth closer inspection:

initial_candidates = client.memories.search(
    query="Does the catalog have the true first printing, not a later state, of this title?",
    retrieval_config=HybridRetrieval(limit=15),
)

For a search this subtle, where correctly distinguishing a genuine first-printing note from a superficially similar entry about a later printing matters enormously to a collector, a reranking step applied specifically to these fifteen candidates can weigh the query against each one far more carefully than the fast initial search alone, correctly promoting the entry that actually addresses the specific bibliographic detail the buyer cares about, rather than one that merely mentions the title in a generally similar way. Running this careful second pass only against this narrowed set of fifteen, rather than against the dealer’s entire catalog, is exactly what keeps this extra precision affordable, applying the more expensive judgment only where the initial fast search has already done the work of narrowing things down to a manageable, worthwhile set.

Reranking improves precision by comparing a query against each candidate individually and jointly. A related but structurally different technique achieves something similar without fully sacrificing the speed of independent, precomputed embeddings, by preserving more detail from each piece of text than a single, compressed vector ever could. Our next chapter, What is late interaction retrieval?, takes up exactly that technique.