Research map / Large language models / Alignment and preferences
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares DPO and PPO for LLM alignment theoretically and empirically, finds DPO sensitive to distribution shift, identifies key PPO factors (advantage normalization, large batch, reference EMA), and shows PPO beats DPO on dialogue and code tasks.
Problem
Whether DPO, a reward-free RLHF method popular on academic benchmarks, is truly superior to reward-based PPO for LLM alignment, and why PPO underperforms on those benchmarks.
Approach
Theoretically shows PPO's reachable policy set is a proper subset of DPO's, yet DPO can produce policies that favor out-of-distribution responses; empirically shows DPO is hurt by distribution shift, and that tuned PPO (advantage normalization, large batch size, reference-model EMA) outperforms DPO on dialogue and code generation.
Key results
- On CodeContest, PPO on CodeLlama-34B reaches 22.4% pass rate at 10@1k on the test set, surpassing AlphaCode 41B (16.4%).
- On SafeRLHF with Llama-2-7B, PPO reaches +1.69 helpfulness and -12.08 harm with 99.5% safety rate, versus DPO's -2.86 helpfulness... and 95.8% safety rate.
- On APPS with CodeLlama-34B, PPO improves introductory pass@5 from 38.6% (SFT) to 44.4%, and DPO-Iter drops to 34.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PPO rather than DPO | Llama2-7B alignment · HH-RLHF test set | OpenAssistant reward: 0.611 -> 0.718 | Helped |
| use PPO rather than DPO-Iter | Llama2-7B alignment · HH-RLHF test set | OpenAssistant reward: 0.678 -> 0.718 | Helped |
| use PPO rather than DPO-Iter | Code Llama 34B · CodeContest Test Set | 10@1k: 3.2% -> 22.4% | Helped |
| add advantage normalization (ablation) | PPO · APPS Intro. | pass@5: 18.0% -> 38.1% | Helped |
| use a large batch (ablation) | PPO · APPS Comp. | pass@5: 1.1% -> 7.5% | Helped |
| add exponential moving average reference-model updates (ablation) | PPO · APPS Comp. | pass@5: 7.5% -> 9.1% | Helped |
| use a large batch (ablation) | PPO · CodeContest pass@1k | pass@1k: 7.7% -> 19.6% | Helped |
| add exponential moving average reference-model updates (ablation) | PPO · CodeContest pass@1k | pass@1k: 19.6% -> 21.4% | Helped |
| use SFT (Safe) instead of SFT (Alpaca) as reference model (ablation) | DPO · SafeRLHF | S.R.: 55.4% -> 71.8% | Helped |
| filter dual-unsafe preference pairs (ablation) | DPO · SafeRLHF | ΔHelp.: -1.62 -> 2.46 | Helped |
| filter dual-safe preference pairs (ablation) | DPO · SafeRLHF | ΔHelp.: -1.62 -> -2.86 | Hurt |
| increase β from 0.1 to 0.2 (ablation) | PPO · HH-RLHF, Llama-7B | OpenAssistant reward: 0.718 -> 0.629 | Hurt |
Surprising findings
- DPO trained on CodeContest preference pairs yields a 0.0% pass rate, with outputs that are meaningless code snippets.
- A small PPO batch size (64) can degrade the SFT model on APPS, while large batch sizes give consistent gains.
Limitations
- Reward model training is not studied in depth; PPO and DPO-Iter depend on reward model quality.
- Code competition experiments use ground-truth test-case rewards rather than a learned reward model.
- Theoretical counter-example is a single stateless case with a single preference pair.
Experimental setup
Datasets: HH-RLHF SafeRLHF APPS CodeContest Alpaca
Benchmarks: HH-RLHF SafeRLHF APPS CodeContest
Metrics: OpenAssistant reward GPT-4 win rate helpfulness delta harmlessness score safety rate pass@k
Baselines: DPO DPO-Iter RRHF PRO SFT AlphaCode 41B CodeRL Beaver
Builds on: DPO PPO DeepSpeed-Chat generalized advantage estimation
Closest papers by meaning
- DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.93)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.92)
- AlphaPO: Reward Shape Matters for LLM Alignment (ICML 2025 · similarity 0.91)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
- Bootstrapping Language Models with DPO Implicit Rewards (ICLR 2025 · similarity 0.91)
- MallowsPO: Fine-Tune Your LLM with Preference Dispersions (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.