Decode is memory-bandwidth-bound, and that single fact is why speculative decoding is the cheapest throughput win an inference team can spend a week on. Every autoregressive step loads the entire model’s weights from HBM to SRAM to produce one token — a 70B model at fp16 is ~140GB of weight traffic for a token that might be a comma. The compute is nearly free; the memory movement is not. Speculative decoding exploits this by making the expensive pass verify many tokens at once, so the weight-loading cost is amortized. The concept is old — Leviathan et al. and Chen et al. both published it in 2023 — and vLLM has shipped it since v0.4.2 (April 2024), so by now “does your serving stack have it on?” is the wrong question. The right questions are about acceptance rate, draft-model choice, and the batch effects that hide in benchmarks.

How it works, and the math that decides whether it helps

You run two models: a small, fast draft (say 1-8B) that generates K tokens greedily, and the target model that takes those K draft tokens plus one true token and processes them in a single forward pass — K+1 token positions of work, still one weight load. Each draft token is then accepted or rejected by the target’s distribution; if the draft is good, most are accepted and you net K tokens of progress for roughly the cost of one. The catch is the acceptance rejection sampling: to produce exactly the target model’s distribution (no distribution shift, so quality is untouched), you accept draft token x with probability min(1, p_target(x) / p_draft(x)), and on rejection you resample from the residual (p - q)+ — the part of the target distribution the draft didn’t capture. Code, in torch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
def verify(draft_ids, target_logits, draft_logits):
    p = target_logits.softmax(-1)
    q = draft_logits.softmax(-1)
    accepted = []
    rng = torch.Generator()
    for i in range(draft_ids.size(0)):
        x = draft_ids[i]
        if torch.rand(1, generator=rng) <= torch.minimum(p[i, x] / q[i, x], torch.ones(1)):
            accepted.append(x)
        else:
            residual = (p[i] - q[i]).clamp(min=0)
            residual = residual / residual.sum()
            accepted.append(residual.multinomial(1, generator=rng))
            return torch.stack(accepted), i + 1
    return torch.stack(accepted), len(draft_ids)

The expected tokens produced per proposal for acceptance rate α and draft length K is (1 - α^(K+1)) / (1 - α). Two numbers to internalize: at α=0.7, K=4, you average ~2.8 accepted tokens per proposal; at α=0.5 it collapses to ~1.9. The whole trade hinges on α and on the draft’s speed — the draft must produce K tokens in a fraction of one target step or the verification math stops working. In our H100 serving fleet, an 8B draft in front of a 70B target with K=4 runs at 1.6-2.3x tokens/s, with the spread driven entirely by task difficulty. Code-heavy generation (low α) lands near 1.4x; boilerplate-heavy prose (high α) approaches the theoretical ceiling. Below about α=0.55 the draft becomes dead weight and you’re better off on plain greedy.

Production wiring and the traps

vLLM’s config is the boring, correct layer: --speculative-model <draft> --num-speculative-tokens 4 --speculative-draft-tensor-parallel-size <n>. Everything after that is engineering judgment:

• The tokenizer must be identical, byte for byte. If draft and target disagree on vocabulary, the p/q ratio in rejection sampling is computed across different token spaces and the whole scheme silently produces garbage. We added a startup assertion comparing tokenizer.get_vocab() on both sides, because the one time we skipped it cost us a week of weird per-request flakiness.
• Co-locate draft and target. The verification step reads draft logits; over a network that’s a round trip per proposed block. Draft and target belong on the same NVLink-connected node, ideally with the draft’s KV cache colocated.
• Watch acceptance in telemetry, not just tokens/s. Tokens/s is the outcome; α is the diagnosable variable. vLLM exports speculative_acceptance_rate; alert on it. When it drops from 0.72 to 0.58 on a new eval set, that’s the signal to swap drafts or shrink K.
• K is a knob, not a constant. We run K=4 for general traffic and K=6 for long-form generation where the draft’s predictions are stronger. Tune it against measured α, not intuition.

The draft-model landscape, as of 2026

The strongest family is EAGLE-style: instead of a separate language model, a lightweight auto-regressive head is trained on the target’s own features to predict the target’s next tokens, which is why acceptance rates reach 0.75-0.85 — the draft is literally simulating the target’s distribution. EAGLE-3 (October 2025) removed the KV-cache transfer to the draft entirely, which was the main latency drag in EAGLE-1/2. For open weights, EAGLE-3-style heads on Llama-class targets are the pragmatic default; n-gram drafts (--speculative-model [ngram] in vLLM) are the zero-training fallback for domains where a statistical draft beats an 8B model, and self-speculative / Medusa-style heads (draft by the target’s own early-exit) remain the choice when you can’t afford a second checkpoint.

What benchmarks hide

Single-stream speculative-decoding benchmarks flatter the technique because they assume one request monopolizing the hardware. Under real concurrency, the draft and target contend for the same SMs and bandwidth, and the gains compress — we’ve seen advertised 2x shrink to 1.3x at high request rates on shared hosts. Measure on your production concurrency, not on a synthetic prompt. And know when to turn it off: for pure-batch (offline scoring) workloads with large per-request prefill dominance, speculative decoding saves almost nothing — the win lives entirely in the decode phase.

Speculative decoding is the rare inference optimization with no quality trade-off: rejection sampling guarantees the output distribution is identical to the target’s, so the benchmark you run is honest. Wire it in, watch α, tune K, and pocket the 1.5-2x.