Research map / Large language models / Alignment and preferences
Risk-aware Direct Preference Optimization under Nested Risk Measure
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ra-DPO, a token-level DPO variant that adds a sequential risk ratio (from nested CVaR/ERM risk measures) to the DPO loss to limit model drift from the reference model.
Problem
DPO-style alignment with KL constraints does not provide tight risk control over deviation from the reference model; risk-neutral token-level methods like TDPO ignore reward distribution beyond the mean.
Approach
Formulates a constrained risk-aware advantage maximization under nested risk measures (CVaR, ERM), converts the Bradley-Terry model into a token-level form with a sequential risk ratio term, and derives a loss equal to DPO loss minus a sequential risk ratio term (with a stop-gradient variant, Ra-DPO2).
Key results
- On AlpacaEval, Ra-DPO2 reaches 55.7 length-controlled winrate versus 52.2 for TDPO2 and 44.7 for DPO (vs gpt4_1106_preview).
- Ra-DPO1 achieves the highest raw winrate of 53.5 on AlpacaEval, above TDPO2 at 52.2.
- Ra-DPO variants are reported to achieve higher reward accuracy and lower sequential KL than TDPO2 on Anthropic HH across CVaR and ERM settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the sequential risk ratio to DPO preference optimization (Ra-DPO-1) | DPO · AlpacaEval | Winrate: 52.2$\pm$ 1.6 -> \textbf{53.5$\pm$ 1.8} | Helped |
| Use Ra-DPO-2 with the sequential risk ratio and stop-gradient baseline | TDPO-1 · AlpacaEval | Lc winrate: 53.0$\pm$ 0.6 -> \textbf{55.7$\pm$ 0.5} | Helped |
| Use Ra-DPO-2 instead of Ra-DPO-1 (ablation) | Ra-DPO · AlpacaEval | Winrate: \textbf{53.5$\pm$ 1.8} -> 52.1$\pm$ 1.8 | Hurt |
| Use Ra-DPO-2 instead of Ra-DPO-1 (ablation) | Ra-DPO · AlpacaEval | Lc winrate: 53.9$\pm$ 0.5 -> \textbf{55.7$\pm$ 0.5} | Helped |
| Use nested risk measures in Ra-DPO-2 | TDPO-2 · Anthropic HH | reward accuracy | Helped |
| Use nested risk measures in Ra-DPO-1 | TDPO · IMDb | sequential KL divergence | Helped |
Limitations
- Gains reported are modest and some AlpacaEval differences fall within reported standard deviations.
- Experiments are limited to small models (GPT-2 Large, Pythia 1.4B/2.8B) and the judge is a 12B open model rather than a stronger LLM judge.
- The derivation relies on assumptions (tree-structured Pb-MDP, sum-decomposable reward) that are only partly justified in the text.
Experimental setup
Datasets: IMDb Dataset Anthropic HH Dataset
Benchmarks: IMDb Dataset Anthropic HH Dataset AlpacaEval
Metrics: reward accuracy sequential KL divergence winrate length-controlled winrate
Baselines: DPO PPO TDPO1 TDPO2 KTO
Builds on: DPO TDPO KTO TRPO
Closest papers by meaning
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.89)
- Preference Optimization by Estimating the Ratio of the Data Distribution (NeurIPS 2025 · similarity 0.89)
- KL Penalty Control via Perturbation for Direct Preference Optimization (NeurIPS 2025 · similarity 0.88)
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.88)
- Token-level Direct Preference Optimization (ICML 2024 · similarity 0.88)
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.