BLEU and ROUGE did not catch a single regression for us. The rewrite that scored +3 ROUGE against the old transcripts was, by any human reading, worse — it hedged, refused, and dropped the product SKU out of answers. n-gram metrics cannot judge “did the customer get a usable answer”, and at our volume — 40 model versions a week across three prompt configs — a human review queue for every candidate was never going to happen. We needed a judge. The hard part turned out not to be the judge model — it was proving the judge was not lying to us before wiring it into the deploy gate.

Don’t trust the judge until you measure the judge

An LLM judge is a measurement instrument, and instruments need calibration. Ours looked perfect on spot checks and was wrong in a systematic way: it favored long, fluent answers regardless of whether they answered the question. We only caught it when we computed agreement against human labels and saw Cohen’s kappa of 0.58 — “moderate” agreement, i.e., roughly a coin flip on the disagreements that mattered.

So the harness calibrates before it gates: we run the judge over a labeled ~200-sample slice with human annotations and compute agreement. Kappa below 0.7 means the judge is not ready — we iterate on the rubric prompt until it clears. Kappa is the right metric because it subtracts chance agreement; with 85% “good” answers, raw accuracy is 0.85 before the judge emits a token:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import numpy as np

def cohens_kappa(a, b, k):
    """Cohen's kappa for k mutually exclusive categories."""
    n = len(a)
    if n == 0:
        return 1.0
    conf = np.zeros((k, k))
    for x, y in zip(a, b):
        conf[x, y] += 1
    p_o = np.trace(conf) / n
    p_e = (conf.sum(axis=0) / n) @ (conf.sum(axis=1) / n)
    return (p_o - p_e) / (1 - p_e + 1e-12)

# Judge vs human on a 200-sample labeled slice, 4 severity buckets
print(round(cohens_kappa(human_labels, judge_labels, k=4), 2))  # e.g. 0.82

We treat 0.8+ as deployable for coarse rubrics; 0.7-0.8 is screening-only; below that, the rubric goes back for iteration.

Judge design: the calibration is in the details

Five decisions moved our kappa more than the choice of judge model:

• Rubric before examples. A judge prompted with “rate 1-5” drifts. We pin explicit anchors — “1 = does not answer”, “3 = answers but omits a required field”, “5 = complete and correct” — because anchors are what make scores reproducible across judge versions.
• Blind to the system. We pass the answer with the system identity stripped. Judges measurably favor outputs they recognize; once we removed model names from the prompt, kappa rose from 0.58 to 0.74.
• Reference-aware. For factual tasks, the judge gets a verified reference answer and scores on (answer, reference) pairs. Reference-less judging is fine for style and safety; it is unreliable for factual correctness.
• Self-consistency voting. We run the judge three times at temperature 0.3 and majority-vote. Judge variance is real — a single call flips ~8% of borderline scores — and voting collapses that to ~2%.
• Coarse buckets, not fine scales. A 1-5 scale underperforms a 4-point rubric with distinct anchors; the middle of a 5-point scale is where human agreement itself collapses.

Which judge model? In mid-2024 the practical floor was a frontier model — GPT-4-class or Claude 3 Opus. We validated empirically rather than trusting the “any small model works” line: our 13B-judge experiments topped out around kappa 0.65 and were rejected. Pin the judge version in config — frontier judges ship changes monthly and a silent update shifts every score.

The gate

Scores without a gate are a dashboard. Ours works like this: every candidate model/config runs the fixed evaluation slice in CI, scores aggregate per rubric dimension, and the deploy blocks if any dimension regresses more than 0.1 on the 0-1 scale versus the incumbent, or if judge-human kappa on the freshly sampled calibration slice dips below 0.75. The gate has caught three regressions ROUGE missed entirely, including the hedgy chatbot: -0.22 on “actionability” at +0.03 ROUGE.

Production lessons

• Judge cost is a line item, budget it. Our 1,200-sample eval slice costs roughly $1.20 in judge tokens per run (3 votes, ~700 tokens per judge call). Running 40 weekly candidates through it is affordable; running it in request-path loops is not.
• The rubric and the judge are one artifact. Version them together. We have “eval_rubric_v7 + judge_model=gpt-4o-2024-08-06” as a single deployable unit, because splitting them is how you get a “regression” that is actually a rubric drift.
• Sample for the tails. Uniform random sampling under-represents rare categories — the exact ones that regress. We stratify the eval slice by intent, forcing in the failure modes that matter (refusals, hallucinated SKUs, missing escalation).
• Watch for judge-content pollution. A judge that has seen the golden answer in training is inflating your numbers. For public benchmarks we keep a private labeled slice the judge’s training corpus almost certainly missed.

An LLM judge is the only evaluation that scales to every prompt tweak, but it is only as good as the kappa you measured, not the one you hoped for. Measure the instrument, gate the deploy, version the rubric with the model — everything else is dashboarding.