Reranking

What is reranking?

Reranking is a second scoring step in a search pipeline. The first search retrieves a broad list of candidates quickly. The reranker then reviews those candidates more carefully and puts the best matches at the top.

In RAG, this usually sits between retrieval and answer generation. A vector database or hybrid search engine finds, for example, the top fifty chunks. A reranker scores each chunk against the user's question. The top three to five chunks then go to the language model.

The purpose is simple: retrieve broadly, then choose precisely.

Why vector search often needs reranking

Vector search uses embeddings to compare meaning. It is fast and good at finding candidates that are roughly related to the question. At large scale, it often uses approximate nearest neighbour search because exact comparison across millions of vectors would be too slow.

That speed comes with a trade-off. The right answer may be in the candidate list, but not in first place. Similar-looking chunks, generic FAQ sections, or repeated boilerplate can crowd out the passage that actually answers the question.

Reranking fixes the order. It does not search the whole knowledge base again. It only re-scores the shortlist returned by the first retrieval step.

How a reranker works

Many first-stage search systems use a bi-encoder pattern. The document is embedded ahead of time, the query is embedded at search time, and the system compares the two vectors. That is fast because document vectors are already stored.

A reranker often uses a cross-encoder pattern. It receives the query and one candidate passage together, then scores how well that exact passage answers that exact query. Because the model sees both pieces at the same time, it can judge relevance more sharply.

The cost is latency. A cross-encoder must run once per candidate. Reranking fifty passages is practical. Reranking five million passages is not. That is why reranking is a second-stage step, not a replacement for retrieval.

Reranking in a RAG pipeline

1. Retrieve broadly. Use vector search, keyword search, or hybrid search to fetch a candidate set.

2. Rerank the shortlist. Score each candidate against the question and sort by relevance.

3. Trim the context. Pass only the strongest chunks to the model, keeping the context window focused.

4. Generate the answer. The language model writes from better evidence and has less irrelevant text to distract it.

For example, a support assistant receives the question Can I pause my subscription for two months? Vector search may retrieve cancellation policy, pricing pages, a generic account article, and the actual pause policy. The reranker can push the pause policy to the top because it compares the full question with each passage directly.

Rerankers and semantic rankers

Reranking can be delivered in several ways. Cohere Rerank is a hosted API. Sentence-Transformers and similar libraries let teams run cross-encoder models themselves. Azure AI Search has semantic ranking features that reorder search results using language understanding. Search platforms may also combine keyword rank, vector similarity, reciprocal rank fusion, and a final semantic ranker.

The naming varies. The design pattern is the same: a cheap first pass finds candidates, and a more expensive second pass sorts them better.

What to watch out for with reranking

Measure retrieval quality. Build a test set of questions with the passages that should be retrieved. Compare recall and rank before and after reranking.

Watch latency. Every extra candidate costs time. Test top 20, top 50, and top 100 instead of guessing.

Use hybrid search where needed. Reranking cannot rescue a candidate that was never retrieved. Exact terms, product codes, names, and legal references often need keyword search alongside embeddings.

Keep chunks readable. A reranker works better when candidates are coherent passages, not random text fragments or half tables.

Do not rerank everything. Use reranking where answer quality matters enough to pay for the extra model call.