Research map / Reinforcement learning / RL from feedback
Doubly Robust Alignment for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DRPO, a doubly robust preference evaluation and optimization method for LLM alignment that stays consistent if either the preference model or the reference policy is correctly specified.
Problem
RLHF algorithms are sensitive to misspecification of the Bradley-Terry preference model, the reference policy, or the reward model, leading to poor fine-tuning.
Approach
Adapts doubly robust estimation from causal inference and bandits to pairwise preference evaluation: combines a direct-method preference model term with an importance-sampling residual correction, and maximizes this DR total-preference estimator minus a KL penalty to fine-tune the LLM.
Key results
- On TL;DR, DRPO (BT) wins against IPO 98.5%, CPO 90.0%, Dr. DPO 72.5%, and ORPO 57.5% in GPT-4o-judged win rates.
- On HH, DRPO (GPM) achieves 86.38% length-controlled win rate and 84.84% win rate vs SFT, comparable to robust DPO variants (Dr. DPO 92.16%, rDPO 86.89%).
- On IMDb, the DR preference estimator with both models misspecified has markedly larger MSE than estimators with either model correct, and the both-correct estimator approaches zero MSE with 1500 samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DRPO-BT instead of the listed DPO variants | Summarization on TL;DR · TL;DR | pairwise win rate: 72.5% vs Dr. DPO; 65.0% vs rDPO; 63.5% vs cDPO; 90.0% vs CPO; 57.5% vs ORPO; 98.5% vs IPO; 69.5% vs RSO | Helped |
| Use DRPO-GPM rather than SFT | Human dialogue · HH | win rate: 84.84% | Helped |
| Use DRPO-GPM rather than SFT | Human dialogue · HH | length-controlled win rate: 86.38% | Helped |
| Use DRPO-BT rather than PPO | Human dialogue · HH | pairwise win rate: 57% | Helped |
| Use DRPO-BT rather than PPO | Human dialogue · HH | in-distribution performance | Helped |
| Use DRPO-GPM rather than other methods | Human dialogue · HH | in-distribution performance | Helped |
| Use DRPO rather than other DPO variants | Human dialogue · AlpacaEval 2.0 | pairwise win rate | Mixed |
| Keep either the preference model or reference policy correctly specified in DR evaluation | DR preference estimator · IMDb | MSE | Helped |
| Correctly specify both the preference model and reference policy in DR evaluation | DR preference estimator · IMDb | MSE | Helped |
| Misspecify both the preference model and reference policy | DR preference estimator · IMDb | MSE | Hurt |
Surprising findings
- DRPO-BT beats PPO with the same reward model only 57% of the time on HH, and the paper notes DRPO does not model HH pairwise noise, so it underperforms robust DPO variants out-of-distribution there.
Limitations
- Theoretical guarantees rely on coverage, boundedness, realizability and VC-class assumptions.
- Preference model used on HH does not account for pairwise noise, so it trails noise-aware robust DPO variants out of distribution.
- Requires estimating the reference policy and preference model; the reference policy is misspecified in the TL;DR setup, and theory assumes independence via sample-splitting.
Experimental setup
Datasets: TL;DR Anthropic Helpful and Harmless IMDb AlpacaEval 2.0
Benchmarks: TL;DR Anthropic HH AlpacaEval 2.0 IMDb
Metrics: win rate length-controlled win rate MSE
Baselines: PPO DPO Dr. DPO rDPO cDPO CPO ORPO IPO RSO
Builds on: DPO IPO general preference model doubly robust estimation PPO
Closest papers by meaning
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.92)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.92)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.92)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.92)
- One-Shot Safety Alignment for Large Language Models via Optimal Dualization (NeurIPS 2024 · similarity 0.92)
- Clone-Robust AI Alignment (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.