Off-the-shelf embedders understand generic language and nothing else. BGE-small and OpenAI’s Ada-002 encode “what is the refund policy” beautifully; ask them about an internal part number like OCT-4472-REV3 or a domain term like “fargate egress draining” and they shrug — the token is out-of-vocabulary-adjacent and there is no training signal that groups it with the right documents. Our internal RAG was living proof: recall@10 on real queries sat around 0.41 because the embedder kept retrieving documents that shared vocabulary but not meaning.

The fix is not a bigger off-the-shelf model. It’s fine-tuning a small embedder on pairs mined from your own traffic, and the discipline is almost entirely about the data, not the training loop.

Where the training data comes from

You already have the labels: query logs and the documents people actually used. We pulled a year of search and RAG traffic and kept only high-confidence pairs — queries followed by a click, a bookmark, or a “this was helpful” signal. That filter is worth more than any loss function.

The rules that kept our eval honest:

1. Split by time, never randomly. Random splits leak future documents into the training set and inflate every metric. We train on months 1-9 and eval on months 10-12.
2. Eval documents never appear in training. Embedders memorize; an eval that shares documents with training will report recall that your real system will never see.
3. Mine hard negatives. For each query we take the BM25 top-20 and rerank with a small cross-encoder (ms-marco-MiniLM); documents that look similar but are wrong become hard negatives. This is what forces the model to discriminate on meaning instead of lexical overlap.

The loss and the batch

MultipleNegativesRankingLoss treats every other query in the batch as a negative for the current one, which means batch size is the number of negatives the model sees. This is the one hyperparameter that moves the needle. We saw roughly +3 nDCG@10 going from batch 32 to 64 on an A10G (fp16); batch 128 via gradient accumulation was flat — the in-batch negatives are the signal, and accumulation can’t manufacture more of them per step.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
from sentence_transformers import (
    SentenceTransformer, InputExample, losses, evaluators
)

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

# each row: query, positive doc, list of hard negatives
rows = load_pairs("pairs_train.jsonl")
examples = [InputExample(texts=[q, pos, *neg]) for q, pos, neg in rows]
dataloader = DataLoader(examples, shuffle=True, batch_size=64)

evaluator = evaluators.InformationRetrievalEvaluator(
    queries=eval_queries, corpus=eval_corpus, relevant_docs=relevant
)

model.fit(
    train_objectives=[(dataloader, losses.MultipleNegativesRankingLoss(model))],
    epochs=3,
    warmup_steps=200,
    evaluator=evaluator,
    evaluation_steps=500,
)

Note the hard negatives are passed as additional texts inside the same InputExample, so the loss sees them as extra negatives per anchor — that’s the syntax that actually gets them into the gradient. Three epochs was the sweet spot; by epoch five the InformationRetrievalEvaluator was rolling over even as training loss fell, which is the classic overfit signature.

What the numbers said

On the held-out (time-split) set, nDCG@10 went from 0.41 to 0.73 after three epochs on ~180K pairs. The gains were concentrated exactly where we wanted them: queries containing product codes and internal nomenclature improved most, while generic queries barely moved. There is a real trade here — domain fine-tuning regresses generic-domain performance, so we ran an MTEB spot-check and accepted a small dip in general retrieval. If your workload is mixed, budget for two embedders and route by query type rather than pretending one model does both.

Serving the fine-tuned model

A 384-d vector at fp32 is 1.5 KB, so the serving story is easy: export to ONNX, serve with ONNX Runtime (or a SentenceTransformer backend), and let Qdrant do the search. Our setup is 5M documents on a single Qdrant node with ~20 GiB RAM and p99 query latency under 8ms — the fine-tuned model is a drop-in replacement for the old embedder because the dimension didn’t change.

The deployment process matters as much as the training:

• Rebuild the index on a schedule, not on demand. We regenerate nightly, shadow the new index for a week, A/B it against the live one, and swap only when the eval set says the new vectors are better.
• Re-run the hard-negative mining after you retrain. The cross-encoder’s notion of “looks similar but wrong” shifts as the embedder improves; the eval set should be re-mined each cycle or your measured gains decay.
• Keep a regression query set — the 50 queries that used to work. Domain fine-tuning occasionally flips a previously-correct neighbor, and a fixed regression set catches that before users do.

Production lessons that held up: the data pipeline is the model; batch size is the training knob that matters; and if your eval set shares documents with your training set, every number you quote is fiction.