Research map / Large language models / Alignment and preferences
Provably Robust DPO: Aligning Language Models with Noisy Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes robust DPO (rDPO), an unbiased de-noised DPO loss under random preference flips, with sub-optimality bounds scaling as 1/(1-2ε) and experiments on IMDb sentiment and Anthropic HH.
Problem
DPO and RLHF degrade when preference labels are noisy (flipped or ambiguous), and existing heuristics such as label smoothing (cDPO) lack theoretical justification; the paper asks how to learn a policy robust to random preference flips with provable guarantees.
Approach
Adapt the DPO binary cross-entropy loss with the known flip rate so that the expected loss under noisy labels equals the clean DPO loss (unbiased estimator), yielding rDPO with gradient weights tuned to the noise level; the paper also proves estimation and sub-optimality bounds scaling with 1/(1-2ε).
Key results
- On IMDb at noise 0.4 (200-1000 steps), rDPO reaches 0.99 mean reward with near-zero std, while DPO fluctuates between 0.72 and 0.93 and cDPO stays around 0.82-0.84.
- On Anthropic HH, rDPO (eps=0.1) improves over SFT by 51.20% on Llama-2-7b versus 45.78% for DPO and 39.16% for cDPO, judged by GPT-4.
- Theory: sub-optimality gap of rDPO scales as O(sqrt(kappa)/(gamma beta(1-2eps)) sqrt(d/n)), so the cost of noise is a factor 1/(1-2eps), i.e. about 1/(1-2eps)^2 more samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use rDPO instead of DPO on noisy preference data | gpt2-large · IMDb, noise level 0.4, 1000 steps | mean reward: 0.88 $\pm$ 0.32 -> 0.99 $\pm$ 0.00 | Helped |
| Use rDPO instead of cDPO on noisy preference data | gpt2-large · IMDb, noise level 0.4, 1000 steps | mean reward: 0.83 $\pm$ 0.37 -> 0.99 $\pm$ 0.00 | Helped |
| Use rPPO instead of PPO on noisy preference data | gpt2-large · IMDb, noise level 0.4 | mean reward: 0.78 $\pm$ 0.41 -> 0.94 $\pm$ 0.23 | Helped |
| Use rPPO instead of cPPO on noisy preference data | gpt2-large · IMDb, noise level 0.4 | mean reward: 0.87 $\pm$ 0.33 -> 0.94 $\pm$ 0.23 | Helped |
| Use rDPO instead of DPO | gpt2-large · Anthropic helpful-harmless | improvement over SFT (%): 22.20 -> 24.32 | Helped |
| Use rDPO instead of cDPO | gpt2-large · Anthropic helpful-harmless | improvement over SFT (%): 18.34 -> 24.32 | Helped |
| Use rDPO instead of DPO | Llama-2-7b · Anthropic helpful-harmless | improvement over SFT (%): 45.78 -> 51.20 | Helped |
| Use rDPO instead of cDPO | Llama-2-7b · Anthropic helpful-harmless | improvement over SFT (%): 39.16 -> 51.20 | Helped |
| Vary the assumed flip rate; best reported setting is $\epsilon = 0.1$ (ablation) | rDPO · Anthropic helpful-harmless | evaluation accuracy | Mixed |
Surprising findings
- cDPO (label smoothing) did not mitigate the effect of noise on IMDb, performing similarly to or worse than vanilla DPO in that setting.
- Vanilla DPO trained on noisy data at eps=0.4 still performs reasonably in some scenarios, suggesting robustness in some regimes.
Limitations
- Assumes the flip rate epsilon is known to the learner (tuned via cross-validation in practice).
- Bounds rely on log-linear policy class and feature coverage assumptions; neural policy bounds are only sketched and the error does not vanish for neural policies unless alpha_2 = 0.
- Noise model is uniform random flipping, not the instance-dependent or ambiguous noise seen in real data; the Anthropic HH noise level is unknown.
Experimental setup
Datasets: IMDb Anthropic helpful-harmless
Benchmarks: IMDb controlled sentiment generation Anthropic helpful-harmless single-turn dialogue
Metrics: mean ground-truth reward win rate vs chosen responses improvement over SFT (%) implicit reward evaluation accuracy
Baselines: DPO cDPO IPO SLiC PPO cPPO
Builds on: DPO conservative DPO (label smoothing) PPO SLiC IPO
Closest papers by meaning
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.95)
- ROPO: Robust Preference Optimization for Large Language Models (ICML 2025 · similarity 0.93)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.92)
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.91)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.91)
- $\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$ (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.