Research map / Large language models / Alignment and preferences
Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Wasserstein DPO (WDPO) and KL DPO (KLDPO), distributionally robust variants of DPO that handle preference distribution shift, with finite-sample estimation bounds for log-linear policies and tractable gradient-based approximations, evaluated on emotion, ArmoRM multi-objective, and OpenLLM Leaderboard alignment.
Problem
DPO-based LLM alignment is fragile under preference distribution shift between static training preference data and the preferences of deployment users.
Approach
Formulate DPO as a min-max problem over Wasserstein and KL uncertainty sets around the nominal preference distribution, derive finite-sample guarantees for log-linear policies, and approximate the inner maximization by gradient regularization (WDPO) and loss-based reweighting (KLDPO) so standard gradient descent can be used.
Key results
- On OpenLLM Leaderboard v2 with LLaMA-3.2-1B, KLDPO (tau=0.05) reaches IFEval 0.56 vs DPO 0.48 at epoch 4 and WDPO (rho=0.005) gets BBH 0.35 vs DPO 0.35 (ties) while several robust variants beat DPO on IFEval.
- For LLaMA-3.1-8B, KLDPO (tau=0.01) reaches IFEval 0.75 vs DPO 0.62 and MMLU 0.37 vs 0.33.
- Theory gives O(n^{-1/4}) estimation error rate for robust policy parameters versus O(n^{-1/2}) for non-robust DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WDPO or KLDPO instead of DPO | GPT-2 emotion alignment · Emotion dataset under fear-to-anger preference shift | emotion reward performance | Helped |
| Use WDPO or KLDPO instead of DPO | LLaMA-3.2-1B-Instruct ArmoRM multi-objective alignment · Five ArmoRM evaluation objectives, including unseen objectives | ArmoRM objective scores | Helped |
| Use robust DPO (WDPO/KLDPO) instead of DPO | LLaMA-3.2-1B/3B and LLaMA-3.1-8B · OpenLLM Leaderboard v2 | benchmark score | Mixed |
| Train DPO for 6 rather than 2 epochs (ablation) | LLaMA-3.2-1B-Instruct · OpenLLM Leaderboard v2 | MATH: 0.08 -> 0.06 | Hurt |
| Train DPO for 4 rather than 2 epochs (ablation) | LLaMA-3.2-1B-Instruct · OpenLLM Leaderboard v2 | MATH: 0.08 -> 0.07 | Hurt |
| Increase WDPO robustness parameter from ρₒ=0.005 to ρₒ=0.01 (ablation) | LLaMA-3.2-1B-Instruct · OpenLLM Leaderboard v2 | MATH: 0.09 -> 0.09 | No clear effect |
| Increase KLDPO temperature from τ=0.05 to τ=0.1 (ablation) | LLaMA-3.2-1B-Instruct · OpenLLM Leaderboard v2 | BBH: 0.36 -> 0.36 | No clear effect |
| Decrease KLDPO temperature from τ=0.01 to τ=0.005 (ablation) | LLaMA-3.1-8B-Instruct · OpenLLM Leaderboard v2 | MATH: 0.22 -> 0.24 | Helped |
| Decrease KLDPO temperature from τ=0.01 to τ=0.005 (ablation) | LLaMA-3.1-8B-Instruct · OpenLLM Leaderboard v2 | MUSR: 0.36 -> 0.34 | Hurt |
Surprising findings
- The robust WDPO estimation error rate O(n^{-1/4}) is worse than the O(n^{-1/2}) of standard DPO, which the authors attribute to the robust setting.
Limitations
- Robust rate O(n^{-1/4}) is slower than non-robust DPO O(n^{-1/2}); open question per authors
- Theory assumes log-linear policy class; neural extension only sketched in a remark
- No error bars or statistical significance reported in experiments
Experimental setup
Datasets: Emotion dataset HelpSteer2 UltraFeedback
Benchmarks: OpenLLM Leaderboard v2 Emotion alignment preference shift ArmoRM multi-objective evaluation
Metrics: accuracy/score on OpenLLM Leaderboard v2 subtasks (IFEval, B reward under mixing-coefficient shift ArmoRM objective scores
Baselines: DPO chiPO GRPO (group robust preference optimization) RLHF reward model with PPO
Builds on: DPO RLHF Bradley-Terry model Wasserstein DRO regularization
Closest papers by meaning
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.95)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.94)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.94)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.93)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.93)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.