What does a reranker do, when is it worth the added latency, and what can it not fix?
Answer it out loud first, then compare. That's what makes it stick.
First-stage retrieval is cheap and approximate: it compares the query to each chunk separately. A cross-encoder reranker reads the query and each candidate together, so it judges relevance much more accurately, but it is too slow to run over the whole corpus. So I retrieve maybe 50 candidates, rerank, and send the top 5 to 10 to the model. It adds some latency, and it can only reorder what retrieval found. If the right passage is not in the candidates, I fix recall.
- Retrieve wide and cheap, rerank narrow and precise
- Cross-encoder reads query and passage together
- Typical shape: 50 candidates down to 5-10
- Cost: extra latency and per-call fees
- Limit: cannot recover what retrieval missed
What it does. A first-stage retriever (vector, BM25 or hybrid) scores each chunk against the query independently, so it is fast but coarse. A cross-encoder reranker feeds the query and one candidate together through a model and outputs a relevance score. It sees the interaction between the two, which is why it ranks better, and why it cannot be run over millions of chunks.
The standard shape.
- Hybrid retrieval returns about 50-150 candidates (high recall).
- The reranker scores them and keeps the top 5-10 (high precision at the top).
- Only those go into the prompt, which also saves tokens and reduces distraction for the generator.
Evidence it helps. In Anthropic's contextual retrieval experiments, contextual embeddings plus contextual BM25 cut top-20 retrieval failures by 49% and adding a Cohere reranker took the reduction to 67% (5.7% down to 1.9%). Managed search products ship rerankers for the same reason; Azure AI Search runs its semantic ranker after RRF fusion and, per its docs, accepts up to 50 results.
What it costs.
- Latency: extra model inference on every query. Measure it on your candidate count; a budget of tens to a few hundred milliseconds is typical for hosted rerankers, so state your own measured figure.
- Money: hosted rerankers charge per search or per document scored. Self-hosted open models trade that for GPU operations.
- Another dependency that can fail or rate-limit, so decide what the fallback is (skip reranking and use the fused order).
What it cannot fix.
- A reranker only reorders candidates. If the right passage never made the candidate set, the fix is recall: chunking, hybrid search, query rewriting, a larger candidate pool.
- It cannot repair a bad parse or missing metadata.
- Generic rerankers can underperform on jargon-heavy domains; test on the customer's questions.
How to decide it is worth it. On a labelled set, compare recall@50 (did retrieval find it?) with precision at 5 or MRR before and after reranking. If recall@50 is high but MRR is low, reranking will help. If recall@50 is low, skip it and fix retrieval first.
Say it this way: hybrid retrieval for recall, reranker for precision at the top, and I measure each on its own.
- Recall@50 is 70%. Will a reranker get you to 90%?
- Latency budget is 2 seconds to first token. Where does the reranker fit?
- Hosted reranker or self-hosted open model for a regulated customer?
- RAG system design interview: a worked FDE answer (prep, 2026-09-25) FDE answer: hybrid retrieval for recall, a reranker for precision at the top, and the limit that a reranker only reorders what retrieval found.
- FDE System Design Interview: Customer Deployments, RAG, Integrations (prep, unknown) Worked FDE design that reranks the top 50 down to 8 passages before generation.
- Contextual Retrieval in AI Systems - Anthropic (official, 2024-09-19) Adding a reranker to contextual embeddings and BM25 reduced top-20 retrieval failure by 67% (5.7% to 1.9%) in their tests.
- Create a Hybrid Query - Azure AI Search (docs, unknown) Semantic ranker takes up to 50 results, so size the hybrid candidate pool to at least 50.