Research map / Large language models / Alignment and preferences
ROPO: Robust Preference Optimization for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ROPO, an iterative preference-alignment framework that alternates noise-tolerant training with a robust regularized loss and loss-based noisy-sample filtering, plus robustness-guided rejection sampling, without external models.
Problem
Preference alignment of LLMs (DPO-style ranking methods) degrades substantially when preference labels contain noise, and existing fixes either only soften noise or rely on external reward/teacher LLMs.
Approach
Formulate learning from noisy preferences as a constrained problem with quality-aware sample weights summing to a retained-sample budget; derive a robust loss (DPO loss plus a weighted regularizer) whose gradient weight decreases for samples with large implicit reward margins, so noisy samples get high loss and can be filtered by loss magnitude; add rejection sampling scored by loss to recover dis
Key results
- ROPO's win rate drops at most 2.4% under 20% noise on UFB, versus 14.4% for DPO and 5.0% for IPO (Mistral-7B).
- Even with no artificial noise, ROPO exceeds DPO by 16.0% on TL;DR and 5.6% on Alpaca Comparison.
- Under 40% noise on TL;DR with Llama-2-7B, ROPO reaches 79.2% win rate versus 35.2% for DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ROPO instead of the strongest baseline | Mistral-7B and Llama-2-7B preference alignment · UFB, Alpaca Comparison, and TL;DR at 0%, 20%, and 40% artificial noise | win rate (%) | Helped |
| Increase artificial preference-label noise | DPO and other preference-alignment methods · UFB, Alpaca Comparison, and TL;DR | win rate (%) | Hurt |
| Add the noise-tolerant regularizer ℓreg to the aggressive DPO loss (ablation) | ROPO on Mistral-7B · UFB, 20% artificial noise | win rate (%): 86.21 -> 87.58 | Helped |
| Add noisy sample filtering to ROPO with ℓreg (ablation) | ROPO on Mistral-7B · UFB, 40% artificial noise | win rate (%): 86.34 -> 88.07 | Helped |
| Add robustness-guided rejection sampling to ROPO with ℓreg and NSF (ablation) | ROPO on Mistral-7B · UFB, 0% artificial noise | win rate (%): 89.44 -> 91.06 | Helped |
| Filter 20% or 30% of samples with the largest losses rather than 10% (ablation) | ROPO · UFB and TL;DR | win rate (%) | Helped |
| Vary α within the tested appropriate range (ablation) | ROPO · UFB and TL;DR | win rate (%) | Mixed |
| Train ROPO for a second epoch rather than one epoch (ablation) | ROPO on TL;DR · 20% and 40% artificial noise | win rate (%): 8%-24% increase in the win rate | Helped |
| Train DPO for additional epochs under artificial noise (ablation) | DPO on TL;DR · 20% and 40% artificial noise | win rate (%) | Hurt |
| Use ℓropo rather than DPO loss to rank noisy versus clean samples | ROPO and DPO on TL;DR · top 20% highest-loss samples | proportion of noisy samples: about 34% -> 60% | Helped |
| Use the noise-tolerant regularizer rather than DPO loss | Preference optimization under symmetric label noise · Theoretical preference probability | difference from clean optimum | Helped |
Surprising findings
- The standard DPO loss cannot separate noisy from clean samples: its loss distributions for clean and noisy samples are similar and the gap narrows during training.
- Using 20% filtering as default; larger filtering ratios discard too much clean data.
Limitations
- Uses a pre-fixed sample filtering ratio, risking over- or under-filtering when the noise rate is unknown.
- Noise is only simulated via random symmetric label flipping on existing datasets.
Experimental setup
Datasets: UltraFeedback Binarized Alpaca Comparison Reddit TL;DR
Benchmarks: AlpacaEval Reddit TL;DR test split (500 queries)
Metrics: win rate vs SFT targets / text-davinci-003
Baselines: DPO IPO rDPO cDPO
Builds on: DPO IPO rDPO cDPO
Closest papers by meaning
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.94)
- Provably Robust DPO: Aligning Language Models with Noisy Feedback (ICML 2024 · similarity 0.93)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.91)
- Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.91)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.