Reranking
After Rank Fusion you have a ranked list, but the ranking is purely mathematical — vector distances and keyword scores — with no understanding of what the user actually meant. Reranking takes that shortlist and passes each candidate to a model that scores it against the original query, applying nuanced relevance judgment the math can't. The model can be a general LLM or a specialized reranker like Cohere's. The tradeoff is cost and latency: it's another step and often another API call before final results.
The load-bearing design point is why not just start with a model for the whole search? — speed and context-window limits. The first-stage search has to scan thousands of embeddings quickly; a model can't read them all cheaply. So the pattern is two-stage: cheap, fast, high-recall retrieval narrows thousands to a refined handful, then the slow, accurate model reranks only that shortlist. You've already offloaded the bulk of the work, so the expensive judgment runs over a small set. This is the same cheap-recall-then-expensive-precision split behind Search-Then-Get and the LLM-as-scorer stance of LLM-as-Judge.
Claims
- Fusion ranking is blind to intent; a rerank model adds nuanced relevance judgment. observation — factual: vector distance + keyword score rank by math, not by understanding the query.
- Rerank only a refined shortlist, never the whole corpus — cheap fast recall first, expensive accurate scoring last. principle — durable: a model can't scan thousands of candidates cheaply, so the two-stage split (fast retrieve → slow rerank) is what makes model-quality relevance affordable at scale.
- Reranking buys accuracy and costs latency plus (often) an extra API call. observation — the explicit tradeoff; adopt it only when the precision gain is worth the added time/cost.
- A reranker can be a general LLM or a specialized model (e.g. Cohere). observation — factual: purpose-built rerankers exist alongside using a general model for the same job.
Related
- Rank Fusion — reranking runs on the fused shortlist fusion produces.
- Hybrid Retrieval — the retrieve stage whose output reranking refines.
- Semantic Retrieval — the first-stage recall reranking sits on top of.
- LLM-as-Judge — an LLM reranker is a relevance judge; same "model scores each item" shape.
- Search-Then-Get — the same cheap-then-expensive staging, applied to context loading.
- Distillate: How AI Agents Search Their Memory — Hybrid Retrieval, in Practice (OpenClaw)