Research map / Large language models / Alignment and preferences
Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically disentangles preference data, learning algorithm (DPO vs PPO), reward model scale/data, and policy training prompts for learning from preference feedback, using Tulu 2 models, and distills a recipe.
Problem
Preference-based learning is applied inconsistently across data, algorithms, reward models and prompts, so the individual impact of each component on downstream LM performance is unclear, especially PPO vs DPO.
Approach
Holds the base SFT model fixed (Tulu 2 13B/70B, Llama 3 8B) and varies one preference component at a time across 14 preference datasets, DPO vs PPO on identical data, reward model scale/data mixture, and policy prompt sources, then distills these into a recipe.
Key results
- PPO outperforms DPO on average across four dataset sources (avg. +0.7 points over 6 categories), with the largest gaps in coding (+2.9) and safety (+2.3).
- Best recipe model Tulu 2+PPO 13B with 70B UltraFeedback RM reaches average 62.8 vs 61.0 for Tulu 2+DPO 13B and 56.8 for the SFT model.
- Using a 70B reward model with GSM train prompts raises GSM accuracy from 46% to 62% for PPO, while larger RMs give marginal gains elsewhere.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PPO instead of DPO with matched preference data | Tulu 2 13B preference training · Preference evaluation suite, average | average: +0.7 | Helped |
| Use PPO instead of DPO with matched preference data | Tulu 2 13B preference training · Coding | category score: +2.9 | Helped |
| Use PPO instead of DPO with matched preference data | Tulu 2 13B preference training · Truthfulness | category score: -2.5 | Hurt |
| Use fine-grained per-aspect preference judgments rather than overall judgments (ablation) | UltraFeedback DPO training · Preference evaluation suite | average: 60.3 -> 61.0 | Helped |
| Use preference data from Chatbot Arena | DPO preference training · Safety | category score: 91.8 -> 67.3 | Hurt |
| Increase reward model size from 13B to 70B (UltraFeedback RM) (ablation) | PPO policy training · GSM8K | accuracy: 53.0 -> 58.0 | Helped |
| Increase reward model size from 13B to 70B (UltraFeedback RM) (ablation) | PPO policy training · All evaluations | average: 62.2 -> 62.8 | Helped |
| Train the reward model on a mixed dataset rather than UltraFeedback alone (ablation) | PPO with a 70B reward model · All evaluations | average: 62.8 -> 61.8 | Hurt |
| Use math-targeted policy prompts with a powerful reward model (ablation) | PPO policy training with a 70B reward model · GSM8K | accuracy: 46% -> 62% | Helped |
| Replace UltraFeedback prompts with mixed math and coding prompts (ablation) | PPO generalist policy training · All evaluations | average: 62.2 -> 61.9 | Hurt |
| Replace UltraFeedback prompts with mixed math and coding prompts (ablation) | PPO policy training with a 70B UltraFeedback RM · Coding | category score: 47.3 -> 48.4 | Helped |
| Use mixed math and coding prompts with a 70B UltraFeedback RM (ablation) | PPO policy training · All evaluations | average: 62.8 -> 62.4 | Hurt |
Surprising findings
- Larger and better reward models (70B, larger mixed data) improve RewardBench and best-of-N but yield only marginal downstream gains outside GSM.
- Mixing in additional math and code prompts does not improve overall performance and can hurt it.
Limitations
- Single model suite (Tulu 2) and two base models; no multilingual evaluation.
- Does not measure computational cost of PPO vs DPO.
- Limited compute restricted dataset subsampling to 60,908 examples.
Experimental setup
Datasets: UltraFeedback HH-RLHF HelpSteer Chatbot Arena 2023 Chatbot Arena 2024 AlpacaFarm Human AlpacaFarm GPT-4 PRM800k SHP-2 StackExchange Nectar Capybara 7k
Benchmarks: MMLU GSM8k Big Bench Hard TruthfulQA HumanEval+ MBPP+ ToxiGen XSTest AlpacaEval 1 AlpacaEval 2 IFEval RewardBench
Metrics: accuracy pass@10 % truthful and informative % toxic F1 winrate best-of-N performance
Baselines: DPO Llama 2 Chat 13B Nous Hermes 13B Vicuna 1.5 13B Tulu 2 13B SFT
Builds on: DPO PPO RLHF Tulu 2 UltraFeedback
Closest papers by meaning
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.91)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.91)
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.90)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.89)
- SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning (ICML 2025 · similarity 0.89)
- Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.