Research map / Large language models / Alignment and preferences
Explicit Preference Optimization: No Need for an Implicit Reward Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EXPO, a family of explicit preference optimization losses (compositional and regression-based) that avoid DPO's implicit reward reparameterization, proving DPO-family losses fail to preserve optimal policies and interpolate suboptimally.
Problem
DPO-based preference optimization methods, which rely on an implicit reward reparameterization, suffer from uniform regularization that cannot preserve good reference behavior while improving bad regions, and from suboptimal interpolation as the trade-off parameter varies.
Approach
Instead of deriving losses from a reparameterized RLHF objective, directly posit explicit losses: a compositional loss (supervised KL on BT-induced preference probabilities plus a cross-entropy-style regularizer toward the reference policy) and a regression loss fitting the BT preference probability to a weighted average of the reference-induced and ground-truth preference probabilities.
Key results
- On Anthropic HH, EXPO variants show significant win-rate improvements over DPO and IPO (reported in figure), with EXPO competitive with SimPO and KTO.
- On AlpacaEval 2 with Llama-3-Base-8B-SFT, EXPO (reg) achieves 14.64 raw win rate and 20.32 length-controlled win rate versus DPO's 14.62 and 16.71.
- In the synthetic bandit test, only EXPO converges to the BT-optimal policy while DPO, IPO converge to the mode (degenerate delta policy).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EXPO (reg) instead of DPO | Llama-3-Base-8B · AlpacaEval 2 | Win Rate (LC): 16.71 -> 20.32 | Helped |
| Use EXPO (reg) instead of DPO | Llama-3-Base-8B · AlpacaEval 2 | Win Rate: 14.62 -> 14.64 | No clear effect |
| Use EXPO (comp) instead of DPO | Pythia 2.8B · Anthropic HH | Normalized Entropy: 0.033 -> 0.036 | Helped |
| Use EXPO (reg) instead of DPO | Pythia 2.8B · Anthropic HH | Normalized Entropy: 0.033 -> 0.035 | Helped |
| Use EXPO (comp) instead of DPO | Pythia 2.8B · Anthropic HH | Self-BLEU: 0.93 -> 0.90 | Helped |
| Use EXPO (reg) instead of DPO | Pythia 2.8B · Anthropic HH | Self-BLEU: 0.93 -> 0.88 | Helped |
| Use EXPO (comp) instead of DPO | Pythia 2.8B · Anthropic HH | Distinct-1: 0.018 -> 0.025 | Helped |
| Use EXPO (reg) instead of DPO | Pythia 2.8B · Anthropic HH | Distinct-1: 0.018 -> 0.023 | Helped |
| Use EXPO (comp) instead of DPO | Pythia 2.8B · Anthropic HH | Distinct-2: 0.29 -> 0.35 | Helped |
| Use EXPO (reg) instead of DPO | Pythia 2.8B · Anthropic HH | Distinct-2: 0.29 -> 0.33 | Helped |
Surprising findings
- DPO and IPO converge to the degenerate mode-only policy in the synthetic bandit setting, even though the ground truth is a non-degenerate BT-optimal distribution.
- With finite pairwise labels (empirical preference probability 1), any f-DPO loss yields minimizers independent of the reference policy, so no interpolation occurs.
Limitations
- Real-world experiments are limited to Pythia-2.8B and Llama-3-Base-8B, with small-scale comparisons and few baselines in main text.
- The unsupervised term and interpolation parameter lambda require tuning; the paper notes SimPO/KTO comparisons use extra hyperparameters that EXPO avoids only partially.
- Diversity and some results are from a single run without reported variance; some reported details are deferred or missing.
Experimental setup
Datasets: Anthropic HH IMDb UltraFeedback UltraChat-200k
Benchmarks: Anthropic HH IMDb AlpacaEval 2 synthetic interpolation and preservation tests
Metrics: win rate vs chosen response length-controlled win rate normalized entropy self-BLEU distinct-n
Baselines: DPO IPO f-DPO (Jensen-Shannon) SimPO KTO SFT
Builds on: DPO IPO f-DPO GPO
Closest papers by meaning
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.93)
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.93)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.92)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.92)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.92)
- Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.