Research map / Large language models / Alignment and preferences
Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes DPO to f-divergence constraints (JS, forward KL, alpha-divergences) via KKT complementary slackness, yielding a closed-form reward-policy mapping that avoids estimating the normalizing constant, and evaluates diversity, alignment, and calibration.
Problem
DPO is equivalent to RLHF only under reverse KL regularization, which induces mode collapse and reduced generation diversity, while other divergences produce reward-policy relations that require estimating an intractable normalizing constant.
Approach
Using KKT complementary slackness, the authors show that for f-divergences with 0 not in dom(f') (reverse KL, forward KL, JS, alpha-divergence with alpha in (0,1)), the reward can be written as beta*f'(pi/pref) plus a y-independent constant, so the constant cancels in the Bradley-Terry loss and the method remains a supervised objective without normalizer estimation.
Key results
- On Anthropic HH, RKL-DPO attains the highest alignment accuracy (67.19) but lowest diversity, while FKL attains the highest diversity (distinct-2 0.210 at temp 1.0) at lower accuracy (54.30).
- On IMDB, f-DPO variants achieve a better divergence-vs-reward frontier than both PPO (reward) and PPO (loss) baselines.
- Theory bounds the ECE difference between two policies by a function of their f-divergence; the JSD bound is 4*sqrt(2*D_JS) in expectation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace reverse KL with forward KL in f-DPO | f-DPO on Pythia 2.8B · Anthropic HH test | alignment accuracy: 67.19 -> 54.30 | Hurt |
| Replace reverse KL with forward KL in f-DPO | f-DPO on Pythia 2.8B · Anthropic HH test | predictive entropy: 12.25 -> 13.01 | Helped |
| Replace reverse KL with forward KL in f-DPO | f-DPO on Pythia 2.8B · Anthropic HH test | Distinct-2: 0.151 -> 0.210 | Helped |
| Use JSD instead of reverse KL in f-DPO | f-DPO on Pythia 2.8B · Anthropic HH test | alignment accuracy: 67.19 -> 66.80 | Hurt |
| Use α-divergence with α=0.5 instead of reverse KL | f-DPO on Pythia 2.8B · Anthropic HH test | Distinct-2: 0.151 -> 0.206 | Helped |
| Use α-divergence with α=0.5 instead of reverse KL | f-DPO on Pythia 2.8B · Anthropic HH test | alignment accuracy: 67.19 -> 61.72 | Hurt |
| Use α-divergence with α=0.3 instead of reverse KL | f-DPO on Pythia 2.8B · Anthropic HH test | Distinct-2: 0.151 -> 0.199 | Helped |
| Use α-divergence with α=0.7 instead of reverse KL | f-DPO on Pythia 2.8B · Anthropic HH test | Distinct-2: 0.151 -> 0.202 | Helped |
| Use PPO (loss) rather than PPO (reward) for JSD, forward KL, or α-divergence | PPO · IMDB-sentiment | reward-divergence frontier | Helped |
| Use f-DPO rather than PPO | Preference fine-tuning · IMDB-sentiment | reward-divergence frontier | Helped |
| Use reverse KL rather than JSD | DPO · MT-Bench | GPT-4 preference evaluation | Hurt |
| Use JSD rather than α-divergence | DPO · MT-Bench | GPT-4 preference evaluation | Helped |
Surprising findings
- Forward KL and JS divergence penalties in PPO's reward cause training instability because their values grow much faster than reverse KL, motivating a separate-loss PPO variant.
- Reverse KL DPO scores higher on Anthropic HH alignment accuracy than JSD DPO but lower on GPT-4 MT-Bench judgments.
Limitations
- Total variation and chi-squared divergences are not covered because 0 lies in their derivative domain.
- Diversity-alignment tradeoff is shown only with a single base model (Pythia 2.8B) and one preference dataset for the main HH results.
- Calibration bound is theoretical and the experiments are on a binary prediction framing of HH; the ECE bound is loose.
Experimental setup
Datasets: IMDB sentiment Anthropic HH
Benchmarks: IMDB sentiment Anthropic HH MT-Bench
Metrics: reward divergence predictive entropy self-BLEU distinct-n expected calibration error win/tie/lose rate
Baselines: PPO (reward) PPO (loss) DPO (reverse KL) RLHF with reverse KL penalty
Builds on: Direct Preference Optimization Reinforcement Learning from Human Feedback f-divergence minimization for language model alignment Proximal Policy Optimization
Closest papers by meaning
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.92)
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.91)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.91)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.91)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.