Research map / Reinforcement learning / RL from feedback
Preference Optimization by Estimating the Ratio of the Data Distribution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bregman preference optimization (BPO), a ratio-matching generalization of DPO using Bregman divergences over the likelihood ratio of preference pairs, plus a scaled Basu's power (SBA) gradient-scaling instance.
Problem
Generalizing DPO so the policy matches the optimal target policy without relying on a learned reward model or partition function, while keeping simplicity and optimality guarantees and improving the fidelity-diversity trade-off of probabilistic DPO extensions.
Approach
Reformulates DPO as likelihood-ratio estimation between data preference ratio and model ratio, then replaces the squared/logistic ratio loss with a general Bregman divergence (BPO) whose tractable form needs only policy and reference log-probs; SBA rescales Basu's power divergence so gradient magnitude matches DPO at initialization.
Key results
- On Llama-3-8B-Instruct, BPO achieves 55.9% length-controlled win rate on AlpacaEval 2, reported as state-of-the-art among Llama-3-8B backbones.
- On Anthropic-HH with Pythia-2.8B, SBA reaches 57.0% win rate vs preferred and 3.010 entropy, outperforming DPO (48.5%, 2.801) on both.
- On Mistral-7B-Base, BPO improves LC win rate from 15.1% to 19.1% when built on DPO, and from 21.5% to 23.7% on SimPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DPO with BPO using SBA | Pythia-2.8B dialogue alignment · Anthropic-HH dialogue generation | win rate vs Preferred (%): 48.5 -> 57.0 | Helped |
| Replace DPO with BPO using SBA | Pythia-2.8B dialogue alignment · Anthropic-HH dialogue generation | Entropy: 2.801 -> 3.010 | Helped |
| Replace DPO with BPO using SBA | Pythia-2.8B dialogue alignment · Anthropic-HH dialogue generation | BLEU: 0.145 -> 0.132 | Helped |
| Replace DPO with BPO using SBA | Pythia-2.8B dialogue alignment · Anthropic-HH dialogue generation | Distinct-1: 0.336 -> 0.340 | Helped |
| Use SBA rather than DPO | GPT-J summarization alignment · TL;DR summarization, temperature 0.25 | win rate vs Preferred (%): 47.0 -> 61.0 | Helped |
| Use SBA rather than DPO | GPT-J summarization alignment · TL;DR summarization, temperature 0.25 | Entropy: 0.276 -> 0.318 | Helped |
| Apply BPO to DPO | Mistral-7B-Base · AlpacaEval2 length-controlled | win rate: 15.1% to 19.1% | Helped |
| Apply BPO to SimPO | Mistral-7B-Base · AlpacaEval2 length-controlled | win rate: 21.5% to 23.7% | Helped |
| Increase SBA lambda from -0.5 (ablation) | SBA · Anthropic-HH dialogue generation | win rate | Helped |
| Increase SBA lambda from -0.5 (ablation) | SBA · Anthropic-HH dialogue generation | Entropy | Helped |
| Replace logistic-regression BPO with SBA for FKL f-DPO (ablation) | f-DPO (FKL) · Anthropic-HH dialogue generation | win rate vs Preferred (%): 34.5 -> 39.5 | Helped |
| Replace logistic-regression BPO with SBA for JS f-DPO (ablation) | f-DPO (JS) · Anthropic-HH dialogue generation | win rate vs Preferred (%): 52.0 -> 55.0 | Helped |
Surprising findings
- Unlike f-DPO and f-PO, which trade win rate against diversity, BPO instances improved both win rate and entropy over DPO.
Limitations
- Evaluated only on autoregressive language models up to 8B parameters; extension to multimodal LLMs or diffusion models left open.
- Optimality guarantee assumes sufficient model capacity, which is not achieved in practice.
- Gains on Arena-Hard and Mistral-7B are less consistent; BPO on SimPO underperforms in one of nine cases.
Experimental setup
Datasets: Anthropic-HH Reddit TL;DR UltraFeedback
Benchmarks: AlpacaEval 2 Arena-Hard Anthropic-HH Reddit TL;DR
Metrics: win rate length-controlled win rate predictive entropy self-BLEU distinct-1
Baselines: DPO f-DPO (FKL) f-DPO (alpha-divergence) f-DPO (JS) f-DPO (chi^2) f-PO (Jeffrey) f-PO (JS) SimPO IPO KTO
Builds on: DPO f-DPO f-PO SimPO Basu's power divergence
Closest papers by meaning
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.91)
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.90)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.90)
- MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge (NeurIPS 2026 · similarity 0.90)
- $\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$ (NeurIPS 2024 · similarity 0.90)
- Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.