A client in EU pharma needs inference that never leaves the data center: weights, prompts, and outputs all stay on-prem, and the model has to be legally auditable. Llama 3 70B Instruct (April 2024) was the strongest open model we could serve on the two nodes we had budget for. But “just shard it” doesn’t work — 70B at fp16 is roughly 140 GB of weights, which rules out a single 80 GB GPU, and the real constraint turns out to be KV-cache capacity under concurrent load, not raw weights.

Do the VRAM math before you buy hardware

We sized from the per-GPU budget: weights / TP_size plus KV cache plus activations plus a fragmentation margin. For Llama 3 70B, KV is 2 (K+V) x 80 layers x 8 KV heads x 128 head_dim x 2 bytes, which works out to ~320 KiB per token of sequence. With --max-model-len 8192, a full sequence needs about 2.5 GiB of KV.

That single number decided our architecture:

• TP=2 on 2 x A100-80G: weights take ~70 GiB per GPU, leaving ~9 GiB per GPU for KV — roughly 12-15 concurrent 8K-token sequences before PagedAttention starts swapping or refusing. Fine for a handful of users, useless as a service.
• TP=4 on 4 x A100-80G: weights drop to ~35 GiB per GPU, freeing ~45 GiB per GPU for KV. Because KV heads are sharded across the tensor-parallel group, that’s several hundred concurrent 8K sequences — enough that concurrency became bounded by the scheduler and --max-num-seqs, not by VRAM.

We benchmarked both before writing the deployment manifests. TP=2 peaked around 600 tok/s aggregate at 16 concurrent streams; TP=4 sustained 1,800-2,200 tok/s at 64 concurrent with a 1.4s time-to-first-token and ~28 tok/s per stream during decode. The throughput difference is mostly headroom: TP=2 simply can’t hold the batch.

The serving stack

vLLM gives us PagedAttention and continuous batching, which matters more for aggregate throughput than tensor parallelism ever will. The launch command we ship to the deployment is deliberately boring:

1
2
3
4
5
6
7
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 8192 \
  --max-num-seqs 64 \
  --kv-cache-dtype fp8 \
  --port 8000

Two flags deserve comment. --kv-cache-dtype fp8 halves the KV footprint per token, which on the 4-GPU box is the difference between “plenty” and “more than plenty” of concurrency; on our H100 node it uses the native FP8 path. --max-num-seqs 64 caps the scheduler’s batch — push it higher and you trade per-request decode latency for aggregate throughput, which is exactly the wrong trade for an interactive API.

The benchmark that changed our hardware

We originally ordered TP=4 on a node whose GPUs were wired through PCIe Gen4 rather than NVLink. The first benchmark was brutal: decode crawled at ~11 tok/s per stream and aggregate throughput fell by more than half. The all-reduce after every transformer layer crossed the PCIe bus, and at TP=4 that’s 80 synchronization points per forward pass. We re-pinned the workload onto an NVLink-connected node and per-stream decode more than doubled. If you cannot guarantee NVLink inside a node, stay at TP=2 and buy more nodes — the communication cost will eat the parallelism gain.

We also benchmarked with a tiny OpenAI-compatible client so the numbers were from the real serving path:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import time
from openai import OpenAI

client = OpenAI(base_url="http://infer.internal:8000/v1", api_key="internal")

def bench(prompt, max_tokens=256):
    t0 = time.perf_counter()
    stream = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3-70B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        stream=True,
    )
    first, n = None, 0
    for chunk in stream:
        if first is None:
            first = time.perf_counter() - t0
        if chunk.choices[0].delta.content:
            n += len(chunk.choices[0].delta.content.split())
    total = time.perf_counter() - t0
    return {"ttft_s": first, "words_s": n / total}

Run that against TP=2 and TP=4 and the TTFT and decode numbers speak for themselves. We keep it in the repo as a smoke test that fires on every deploy.

Production lessons

• Pin --max-model-len to what your workload actually uses. Jumping to 16K context halves the concurrency ceiling for a capability almost nobody consumes.
• Use PagedAttention’s scheduler, not TP, to control concurrency: --max-num-seqs is the knob you tune week to week.
• Run DCGM or nvidia-smi loops for thermal and ECC events, and configure vLLM’s /health endpoint into the load balancer. A GPU that silently throttles is worse than a dead one.
• Budget for KV cache, not weights, when deciding how many GPUs to buy. Weights scale linearly with TP; concurrency scales with the leftovers.
• Keep the model warm. Cold-start on 70B is minutes, and model eviction to save VRAM is a false economy at this size.