Reranking — Improving Retrieval Quality

Bi-encoder retrieval (embedding similarity) is fast but approximate — it encodes query and document independently, losing cross-attention between them. Result: the top-k retrieved chunks often include irrelevant passages. Solution: two-stage pipeline: 1. Retrieve top-50 candidates fast with bi-encoder (ANN) 2. Rerank top-50 with a cross-encoder to get true top-3

A cross-encoder takes (query, document) as a SINGLE input and outputs a relevance score. Unlike bi-encoders, it applies full attention between query and document tokens — much more accurate. Trade-off: Cross-encoders are 100× slower than bi-encoders — only use them on a small candidate set (top-20 to top-50).