TL;DRDPO aligns models directly from preferences — simpler than RLHF, no separate reward model.
Direct Preference Optimization skips the reward-model + RL loop. Given pairs of (preferred, rejected) responses, DPO directly adjusts the model to make preferred answers more likely and rejected ones less likely, via a clean loss function. It's simpler, more stable, and cheaper than RLHF while achieving comparable alignment — which is why it (and variants) became popular for open models.
Key points
Optimize directly on preference pairs
No separate reward model or RL loop
Simpler, more stable, cheaper than RLHF
Popular for aligning open models
Common mistakes
Assuming DPO needs an RL setup (it doesn’t)
Poor preference data → poor alignment
Treating DPO and RLHF as identical
Try it: Contrast DPO and RLHF in pipeline complexity.
Example code
<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>RLHF: rankings → reward model → RL (PPO)
DPO: preference pairs → direct loss → done
simpler, stabler</pre></body></html>