Full-parameter fine-tuning of a 7B model needs about 112 GB of VRAM: fp16 weights (14 GB) plus gradients, fp32 Adam moments, and fp32 master weights add up to roughly 16 bytes per parameter, and 70B pushes that past 1 TB. That is why the standard answer for “adapt this model to my domain” is no longer full fine-tuning — it is a rank factorization applied to a frozen base, which turns a multi-node job into a single-GPU one. QLoRA (May 2023) closed the last gap by quantizing the frozen base, and it is now the default starting point for nearly every fine-tune we run.

The LoRA wager

LoRA’s bet is that the weight update ΔW is low-rank: instead of learning a dense d_out × d_in delta, learn B @ A where A is r × d_in, B is d_out × r, and r is tiny (8–64). The adapted forward pass is W·x + (α/r)·B·A·x. Because the base weights stay frozen, trainable parameters collapse to r · (d_in + d_out) per targeted layer — on LLaMA-2-7B with r=16 across attention and MLP projections, that is roughly 4M parameters out of 7B, about 0.06%. No Adam state, no gradients for the base, no copies of the base weights.

1
2
3
4
5
6
7
8
9
10
11
12
13
import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base, r=16, alpha=32):
        super().__init__()
        self.base = base                            # frozen nn.Linear
        self.A = nn.Parameter(torch.randn(base.in_features, r) * 0.02)
        self.B = nn.Parameter(torch.zeros(r, base.out_features))
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A @ self.B)

B is initialized to zero so the adapter starts as the identity (no training-time regressions), and A is small so the rank basis is roughly orthonormal. The α/r scaling is why you see “alpha = 2r” as the house rule: it fixes the initial adapter magnitude to a constant regardless of rank. We treat α/r as a learn-rate proxy — at r=16, α=32 is a fine start; when we push r=64 for harder tasks like code, we scale α along with it.

QLoRA: freeze it harder

QLoRA keeps the LoRA machinery but quantizes the frozen base to 4-bit NormalFloat (NF4), a 16-level codebook densest near zero, with per-64-weight-block absmax scaling. Two details make it trainable rather than just compact:

• Double quantization. The block scales themselves are quantized (to 8-bit), recovering ~0.37 bits per parameter — about 320 MB on a 7B — which goes straight into batch size.
• Paged optimizers. Adam states are offloaded to CPU memory and paged back during the optimizer step, so peak VRAM stays dominated by weights + activations, not optimizer.

The QLoRA paper’s memory table is the pitch: a 7B fine-tune needs 112 GB fully dense, 16 GB with LoRA, and 5 GB with QLoRA; a 65B fine-tune — 780 GB dense — fits on a single 48 GB A6000 with QLoRA at 41 GB. That is the difference between “we need a cluster” and “we need one workstation we already own.”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from transformers import BitsAndBytesConfig, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype="bfloat16",   # keep matmul in fp16/bf16, not 4-bit
)

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", quantization_config=bnb_config)
model = get_peft_model(model, LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
))

Two settings in that config are load-bearing. compute_dtype="bfloat16" keeps the matmuls in fp16/bf16 while only the stored weights are 4-bit — train in 4-bit and quality tanks. And target_modules covers every linear layer including the MLP; the old “attention-only” habit leaves quality on the table because the feed-forward blocks hold most of the factual knowledge.

Production lessons

• Rank is a search hyperparameter, not a given. We sweep r over {8, 16, 32, 64}; for instruction-following r=16 is usually enough, while domain-format tasks (our SQL-generation fine-tune) needed r=32 to stop leaking formatting errors. Over-ranking doesn’t help and just slows training.
• Never serve the 4-bit weights directly for latency-critical paths. Merge the adapter into the base at serving time (fp16), or you pay NF4 dequantization on every forward. We merge and hot-swap; merging is cheap, re-serving is not.
• Eval for catastrophic forgetting, not just task lift. Adapters are surgical, but we have still seen 3% drop on the base-model eval set after aggressive adaptation. Run the base eval every checkpoint or you ship a model that forgot how to answer anything it wasn’t fine-tuned on.
• Double quantization is free money on 7B+. The 0.37 bits/param isn’t huge, but on a 70B it’s ~3.2 GB — and it costs nothing in quality. Turn it on.

LoRA and QLoRA did not make fine-tuning easy; they made it cheap enough to do repeatedly, and that is the property that matters. A fine-tune you can run on one GPU is a fine-tune you will actually iterate on.