We shipped our first RAG pipeline with a single dense retriever and got paged within a week: users searching for a firmware error code like ERR_0x84F2 or a part number PSX-2416-B were getting semantically adjacent but useless results, while the exact string sat five rows down. Dense embeddings are excellent at meaning and terrible at exactness; BM25 is the mirror image. The fix was not to pick one and pray. It was to stop pretending the two score spaces are comparable, fuse on ranks, and let a cross-encoder make the final call.

Where single-retriever retrieval breaks

On our 2M-document support corpus we measured dense-only recall@20 at 0.87, which sounds fine until you segment it: for queries containing a literal identifier, recall collapsed to 0.58. Vectors don’t care about _, case, or digit runs, and ada-002 embeddings at 1536 dimensions actively push near-duplicate strings into distant clusters. BM25 handled identifiers trivially but returned junk for paraphrased questions (“how do I stop the fans ramping up” matched “fan” documents and nothing else).

So we index every chunk twice: once through the embedding model, once through the tokenizer-backed BM25 field, keeping the cleaned text byte-identical on both paths.

Why you cannot blend raw scores

The naive hybrid is a weighted sum of cosine similarity and BM25 score. It fails because the scales drift independently: BM25 scores grow superlinearly with corpus term statistics while cosine is bounded in [-1, 1]. A weight tuned last month is wrong this month. The stable answer is Reciprocal Rank Fusion (RRF, Cormack et al., 2009), which throws away magnitudes entirely and only counts positions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import hashlib
import numpy as np
from sentence_transformers import CrossEncoder

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")

def search(query, dense_top=100, bm25_top=100, rerank_top=50):
    dense = dense_retrieve(query, k=dense_top)      # HNSW over ada-002 vectors
    sparse = bm25_retrieve(query, k=bm25_top)       # same cleaned text, BM25 field
    candidates = rrf([dense, sparse], k=60)[:rerank_top]
    scores = reranker.predict([(query, doc_text(d)) for d in candidates])
    return [d for d, _ in sorted(zip(candidates, scores),
                                 key=lambda x: x[1], reverse=True)[:5]]

On our corpus, RRF (k=60) lifted hybrid recall@20 to 0.93 versus 0.87 dense-only and 0.81 BM25-only. The rank-only formulation also made the fusion immune to one retriever returning garbage magnitudes after an embedding model swap.

The rerank stage

Fusion finds the right neighborhood; the cross-encoder makes the cut. RRF output is a ranked bag of likely matches, so we send the top 50 to a cross-encoder that scores each (query, passage) pair jointly. That joint scoring is what closes the gap on polysemy and on queries where a single keyword dominates the BM25 ranking.

The cost is real: the MiniLM-12 cross-encoder runs at roughly 12 ms/pair on a T4 and 2 ms on an A10, so reranking 50 candidates costs about 100 ms CPU-bound or 15 ms on GPU. Our end-to-end p95 went from 45 ms (dense-only) to ~90 ms with rerank, which we absorbed by capping candidates at 50 and moving the reranker onto a small GPU pool shared across replicas.

Production notes and war stories

• Cache the reranker output. Key by hash of (query, candidate_id) with a 24-hour TTL. Our ops traffic is dominated by a few hundred repeated query templates; cache hits alone cut GPU rerank load by roughly 30%.
• Text cleaning must be identical on both paths. We originally cleaned HTML and normalized unicode only in the BM25 path; dense vectors were computed from raw text. The two retrievers were indexing different documents. Pin one clean() and call it before tokenization and before embedding.
• Cap the rerank window explicitly. Sending the whole fused list to the cross-encoder is the classic regression; every extra candidate costs linear GPU time for sub-linear recall gains past ~100.
• Deduplicate near-duplicate passages. Support docs get mirrored across regions and minor versions; without exact-normalized dedup the same fact occupies three rerank slots.

Alternatives worth knowing

Hybrid-plus-rerank is one rung on a ladder. If your bottleneck is latency, late-interaction models (ColBERT v2) give token-level evidence at near-bilinear cost and skip the two-stage pipeline. If your corpus is dominated by synonyms and rare vocabulary rather than identifiers, a learned sparse model (SPLADE-style) often outperforms BM25 in the sparse leg and needs no manual field tuning. Start with BM25 + dense + RRF + cross-encoder — it is the cheapest architecture that is genuinely hard to beat, and every upgrade we considered above it (learned sparse, late interaction) still keeps the same rank-fusion and rerank spine.