Research map / Large language models / Alignment and preferences
What Matters in Data for DPO?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic theoretical and empirical study showing that, for DPO, chosen-response quality dominates performance while rejected-response quality matters less, and that online DPO with fixed chosen responses reduces to SFT.
Problem
Which characteristics of preference data (chosen vs rejected quality, contrastiveness/preference gap, on-policy data) most determine DPO performance, which remains poorly understood.
Approach
Derives that the DPO optimum reweights the reference policy by the ratio of chosen to rejected distributions, so chosen quality bounds performance; shows online DPO with fixed chosen samples approximates SFT on chosen data, and validates with controlled Tulu-3-8B-SFT experiments varying chosen vs rejected quality.
Key results
- Fixing rejected responses at the lowest tier and raising chosen quality improves LC-AE2 from 15.2 (Low/Worst) to 20.9 (Best/Worst) on Open Assistant 2 and from 25.8 to 36.5 on UltraFeedback.
- Online DPO with fixed best chosen responses nearly matches continual SFT (e.g., UltraFeedback LC-AE2 37.6 vs 35.8).
- Chosen-quality gains (+7.1 to +8.7 LC-AE2) outweigh preference-gap gains (+3.0 to +4.6) in controlled UltraFeedback experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use higher-quality chosen responses instead of low-quality chosen responses while holding the preference gap constant (ablation) | DPO on UltraFeedback · LC-AE2 | score: +8.7 | Helped |
| Use higher-quality chosen responses instead of low-quality chosen responses while holding the preference gap constant (ablation) | DPO on UltraFeedback · LC-AE2 | score: +7.1 | Helped |
| Widen the preference gap while holding chosen quality high (ablation) | DPO on UltraFeedback · LC-AE2 | score: +4.6 | Helped |
| Widen the preference gap while holding chosen quality low (ablation) | DPO on UltraFeedback · LC-AE2 | score: +3.0 | Helped |
| Alter the rejected responses to reduce the gap while keeping high-quality chosen responses identical (ablation) | DPO on UltraFeedback · LC-AE2 | score: +1.4 | Hurt |
| Alter the rejected responses to expand the gap while keeping high-quality chosen responses identical (ablation) | DPO on UltraFeedback · LC-AE2 | score: +0.8 | Helped |
| Use online rejected responses instead of SFT-only training on the fixed best chosen responses | Online-DPO versus Continual SFT · Open Assistant 2 | LC-AE2: 18.7 -> 19.0 | No clear effect |
| Use online rejected responses instead of SFT-only training on the fixed best chosen responses | Online-DPO versus Continual SFT · UltraFeedback | LC-AE2: 35.8 -> 37.6 | No clear effect |
| Add 10% on-policy data to the high-quality UltraFeedback mixture (ablation) | DPO on UltraFeedback · LC-AE2 | score: 34.5 -> 39.4 | Helped |
| Add 20% on-policy data to the low-quality UltraFeedback mixture (ablation) | DPO on UltraFeedback · LC-AE2 | score: 25.8 -> 27.4 | Helped |
| Vary rejected-response quality while keeping chosen responses fixed at Best (ablation) | DPO on Open Assistant 2 and UltraFeedback · GSM8K | accuracy | No clear effect |
| Increase chosen-response quality from Low toward Best while keeping rejected responses Worst (ablation) | DPO on Open Assistant 2 and UltraFeedback · LC-AE2 | score | Helped |
Surprising findings
- Varying rejected-response quality with fixed best chosen responses yields non-monotonic performance, so rejected quality is a weak predictor of DPO outcome.
- Widening the preference gap with chosen quality held fixed yields only +0.8 to +1.4 LC-AE2 gains.
Limitations
- Theory relies on the Bradley-Terry assumption and simplifying independence assumptions for chosen and rejected samples.
- Quality is measured by a reward model (Skywork-Reward-Gemma-2-27B) whose calibration limits may affect conclusions.
- Experiments are limited to one 8B base model and two prompt datasets.
Experimental setup
Datasets: OpenAssistant 2 UltraFeedback Tulu-3 UltraFeedback variant
Benchmarks: AlpacaEval-2 (length-controlled) MMLU IFEval TruthfulQA GSM8K
Metrics: LC win rate accuracy
Baselines: SFT baseline Continual SFT Best-vs-random Best-vs-worst
Builds on: DPO RLHF Bradley-Terry model
Closest papers by meaning
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.94)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.93)
- $\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$ (NeurIPS 2024 · similarity 0.92)
- Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization (ICLR 2025 · similarity 0.92)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.92)
- TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.