Preference Optimization (DPO)

Stage 3 aligns the model to preferences rather than a fixed target. Direct Preference Optimization skips training a separate reward model: it optimizes the policy directly from pairs of (chosen, rejected) completions, anchored to a frozen copy of the SFT model.

Stage summary

Script: dpo.py  ·  Loads: ckpt_sft.pt  ·  Produces: ckpt_dpo.pt  ·  Run: python dpo.py [minutes] [n_pairs] (defaults 10, 2000)

Where the preferences come from

There are no human labels here. Preference pairs are generated by the model itself and scored with cheap programmatic rewards:

Reward	What it measures
sentiment_reward	positive vs negative words, normalized to ~[-1, 1]
required_words_reward	fraction of the prompt's Words: list present (prefix match, so "jump" counts "jumped")
length_penalty	distance from a ~70-word target

compute_reward(mode=...) combines them; the default "both" mode is 0.1·length_penalty + sentiment + 0.5·required_words.

Building pairs

For each prompt, sample k = 4 completions from the current policy, score them, and pair the best against the worst:

out = model.generate(prompts.repeat_interleave(k, dim=0), ...)  # k samples/prompt
(lo, worst), (hi, best) = scored[0], scored[-1]
if hi - lo > 1e-6 and best and worst:
    pairs.append((pids, best, worst))    # ties carry no signal → skipped

Pairs are refreshed from the latest policy every regen_interval = 250 steps, so the model keeps learning from its own improving samples.

The DPO loss

Push the policy to rank chosen above rejected, measured relative to the frozen reference:

\[ \mathcal{L}_{\text{DPO}} = -\log \sigma\!\Big( \beta \big[(\log \pi_\theta(y_w) - \log \pi_{\text{ref}}(y_w)) - (\log \pi_\theta(y_l) - \log \pi_{\text{ref}}(y_l))\big] \Big) \]

where \(y_w\) is the chosen completion, \(y_l\) the rejected one, and \(\pi_{\text{ref}}\) the frozen SFT model.

margin = beta * ((pi_c - ref_c) - (pi_r - ref_r))
loss   = -F.logsigmoid(margin).mean()

• beta = 0.3 controls how hard the margin is pushed relative to the reference (a KL leash — too high and the model drifts off the SFT manifold).
• Dropout is switched off everywhere so log-probs are deterministic.
• Log-probs are summed over completion tokens only (prompt and padding are masked out), since logits at position t predict token t+1.

pref_acc — the fraction of pairs with a positive margin — is the quantity to watch climb during training.

Hyperparameters

Knob	Value
Learning rate	1e-5 (small — alignment, not relearning)
Batch size	16 pairs per update
Samples per prompt (k)	4
Generation length / temperature / top-k	80 / 1.0 / 50
Reward mode	both

python dpo.py             # 10 min, 2000 pairs
python dpo.py 20 4000     # longer run, more pairs

Try the reward knobs

Switch reward_mode to "sentiment" or "words" to steer the model toward cheerier stories or stricter keyword coverage, then compare on the Evaluation page.

---

📄 Source: dpo.py