Quantization is how a 7B model becomes a weekend desktop project instead of a dedicated server: fp16 weights are 14 GB, int8 is 7 GB, 4-bit is about 3.5 GB. But the three toolchains we run in production — GGUF’s k-quants, AWQ, and bitsandbytes’ NF4 — encode very different bets about what you can throw away. Naive round-to-nearest (RTN) makes perplexity crawl, especially because a handful of outlier channels carry a disproportionate share of information, and RTN flattens precisely those. Every serious scheme exists to protect that long tail of outliers.

What each toolchain actually does

GGUF (llama.cpp) splits weights into fixed-size blocks and stores each block with its own scale. The k-quants go further: a Q4_K_M super-block of 32 weights holds one 6-bit shared scale, with 16 weights at 4 bits and the remaining 16 at 2 bits, so the model pays more bits where the value distribution needs them. That asymmetry is why Q4_K_M at ~4.1 GB on LLaMA-2-7B stays within a fraction of a perplexity point of fp16 — and why it runs happily on CPU with only the matmuls offloaded.

AWQ (Activation-aware Weight Quantization) is the opposite strategy: instead of better codecs, it finds the 1% of channels that activations actually light up and protects them. For each layer it searches a small grid of per-channel scaling factors s, choosing the one that minimizes reconstruction error of the output, then quantizes W · s / max(|W · s|) and folds 1/s back into the preceding layer. Because the heavy channels keep most of their precision, AWQ hits near-fp16 quality at 4 bits and, being pure weight quantization, keeps the GEMMs fast on GPU.

NF4 + double quantization (bitsandbytes) is a training tool, not an inference one — it is the backbone of QLoRA. NF4 is a fixed 16-level codebook derived from the quantiles of a standard normal (weights are approximately normal after initialization), so the bins are densest near zero where most weights live. Each 64-weight block is scaled by its absolute maximum; double quantization then quantizes those block scales again, shaving ~0.37 bits per parameter (~320 MB on a 7B) that goes straight back into batch size.

The NF4 math in code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import numpy as np

# 16 levels = quantiles of N(0,1), from the bitsandbytes paper
NF4 = np.array([
    -1.0, -0.6961928, -0.5250731, -0.3949175, -0.2844414, -0.1847734, -0.0910500, 0.0,
     0.0795803, 0.1609302, 0.2461123, 0.3379152, 0.4407095, 0.5626170, 0.7229567, 1.0,
])

def quantize_block(x, block_size=64):
    # per-block absmax normalization, then nearest codebook entry
    x = x.reshape(-1, block_size)
    scale = np.abs(x).max(axis=1, keepdims=True)
    codes = np.argmin(np.abs(x / scale - NF4[None, :, None]), axis=1)
    return codes, scale.squeeze(-1)

def dequantize_block(codes, scale):
    return scale[:, None] * NF4[codes].astype(np.float32)

The absmax per block is the whole trick: it keeps the codebook fixed and lets the scale absorb every distribution shift, which is what makes 4-bit gradients-plus-weights stable enough to train through. The same quantize/dequantize shape — per-block scale, nearest code — is the shared skeleton of every scheme here; they differ only in what the scale is chosen to protect.

Numbers from our fleet

We run AWQ-4-bit Llama-2-7B and 13B on A10s for internal chat. Against fp16 the AWQ models held WikiText-2 within ~0.1 perplexity points, where RTN on the same 4-bit budget drifted 0.5–1.5; more importantly, decode tokens/sec rose about 3x because memory bandwidth, not FLOPs, is the binding constraint on autoregressive generation. GGUF Q4_K_M on a Mac Studio serves LLaMA-2-13B at interactive speed with the 7.8 GB file fully in unified memory. NF4 appears only during QLoRA training, then we merge and unquantize before serving.

Production lessons

• Never evaluate a quantized model on a single benchmark. Perplexity hides task-level cliffs; we have seen 4-bit models lose 3% on exact-match extraction while holding ppl. Always run your top three eval sets before promoting a quantization.
• AWQ needs a calibration set that matches your traffic. The scales are chosen from activations, so calibrating on English fiction and serving product-support queries silently corrupts the protected channels. Recalibrate per application, not per model.
• Don’t use NF4 for inference. It is a training-time transport format; for serving, keep weights in the quant format your kernel is optimized for (k-quants for llama.cpp, AWQ/INT8 for TensorRT/ExLlama-style stacks).
• Quantize, then retest the tail. Error codes, numeric tokens, and math reasoning degrade disproportionately at 4-bit; if your app is heavy on any of those, budget for 8-bit or fp16 on those pathways.