Sijin T V
Sijin T V A passionate Software Engineer who contributes to the wonders happenning on the internet

Direct Preference Optimization (DPO) vs. PPO: A Pragmatic Comparison

Classic RLHF is a four-model circus: a frozen reference policy, a trainable actor, a reward model, and a critic network, plus a KL term to keep the actor honest. It is the most fiddly training setup I have ever run — reward hacking, critic-collapse, and learning-rate knife-fighting are the default state, not the exception. Direct Preference Optimization (Rafailov et al., May 2023) does not just simplify this; it removes the reward model from the loop by deriving the optimal policy in closed form from the preference data itself. We have trained with both, and DPO is now our default for most alignment work — with specific exceptions where PPO stays.

What DPO really is

The derivation is the whole story. DPO observes that the KL-constrained RL objective has a closed-form optimum, π* ∝ π_ref · exp(r(x,y)/β), which can be rearranged to solve for the reward r in terms of the policy π and reference π_ref. Substitute that implicit reward back into the Bradley-Terry preference model and the reward model disappears entirely: you are left with a binary cross-entropy loss over (prompt, chosen, rejected) triples, comparing log-likelihood ratios of the trainable policy against the frozen reference.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

import torch.nn.functional as F

def dpo_loss(policy_chosen, policy_rejected, ref_chosen, ref_rejected, beta=0.1):
    # implicit reward: log(policy/ref) ratio per completion
    chosen = (policy_chosen - ref_chosen).float()
    rejected = (policy_rejected - ref_rejected).float()
    return -F.logsigmoid(beta * (chosen - rejected)).mean()

def dpo_step(batch):
    ...
    loss = dpo_loss(
        logps(batch["chosen"], model), logps(batch["rejected"], model),
        logps(batch["chosen"], ref),  logps(batch["rejected"], ref),
    )
    return loss

The operational payoff is concrete. On a 7B model in fp16: PPO needs policy + reference + reward + critic, about 56 GB of weights plus optimizer states, which does not fit comfortably on a single 80 GB GPU with activations. DPO needs policy + frozen reference — 28 GB of weights — and fits with room to spare. Training is also faster and more stable because every loss step is a standard cross-entropy on a fixed dataset; there is no reward model drifting under your feet. Zephyr-7B-β showed what that buys: a 7B DPO-tuned model scoring 7.34 on MT-Bench, ahead of Llama-2-70B-chat, trained with a few days of A100 time.

Where it goes wrong

DPO’s single biggest footgun is the reference model. The loss is relative — it rewards the ratio of policy to reference — so a reference that drifts (say, you accidentally keep the policy and reference on the same checkpoint and let both update) collapses the signal into gibberish. Freeze it, and freeze it before the first step. We assert ref weights are untouched in our training harness.

The second footgun is the dataset. Duplicate (prompt, chosen, rejected) rows where chosen ≈ rejected produce zero-gradient noise; dedupe before training. And DPO is famously vulnerable to verbosity bias — models learn that long answers win, because long answers do win in human preference data. We filter by length deciles and track mean output length as a training metric.

The third is β. It is a KL-trade-off knob: high β keeps the policy glued to SFT, low β lets it chase preferences aggressively. 0.1 (TRL’s default, and Zephyr’s) is a sane starting point. Too-low β (roughly < 0.05) is where we have seen repetition and mode-collapse; the “explode into gibberish” failure is more about data and reference-model hygiene than the exact β.

When PPO still wins

DPO is offline: it consumes a static preference dataset and can never see its own behavior, so it cannot learn from feedback on actions it actually takes. We keep PPO for:

  • Online feedback loops — tool-use agents where the reward is an execution result, not a human judgment.
  • Multi-step tasks where credit assignment spans several turns.
  • Controllable generation where you need a live reward signal to steer mid-conversation.

The architecture we now run is layered: SFT → DPO on high-quality static preference data to get 90% of the alignment gain at a fraction of the cost, then PPO (or online variants) only for the specific product surfaces that need self-correction. Start there, and you will rarely touch a critic network again.

comments powered by Disqus