Research map / Large language models / Alignment and preferences
Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an energy-based Infinite Preference Model with a unique MLE and the EPA contrastive loss, which adds weak (mismatched-prompt) negatives to DPO-style pairs, and reports gains over DPO on AlpacaEval 2.0 and MT-Bench with Mistral-7B.
Problem
DPO's ideal loss is the Bradley-Terry MLE, which may have multiple minimizers over an infinite response space, so the slope-1 linearity needed to reach the RLHF optimum is not guaranteed for offline alignment.
Approach
Model preference as an energy-based Boltzmann distribution over responses (Infinite Preference Model) whose MLE is unique and equals the RLHF optimum, and approximate its MLE with a contrastive negative-log-softmax loss (EPA) using strong negatives plus free weak negatives drawn from other prompts.
Key results
- EPA (UF-all, 1:3:8 setting) reaches 24.54 / 23.75 on AlpacaEval 2.0 versus DPO's 17.43 / 15.24 on UF-binarized.
- On UF-binarized, EPA (1:1:2) reaches 19.20 / 19.26 on AlpacaEval 2.0 and 7.71 on MT-Bench, above DPO's 7.55.
- With the on-policy weight and margin tricks, EPA reaches 23.00 / 22.47 on AlpacaEval 2.0 versus 22.33 / 20.61 for the best BTM variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DPO with EPA on UF-binarized | mistral-7b-sft-beta · Alpaca-Eval 2.0 | win-rate (length controlled / raw): 17.43 / 15.24 -> 19.20 / 19.26 | Helped |
| Replace DPO with EPA on UF-binarized | mistral-7b-sft-beta · MT-Bench | score: 7.55 -> 7.71 | Helped |
| Replace DPO-PL with EPA on UF-all | mistral-7b-sft-beta · Alpaca-Eval 2.0 | win-rate (length controlled / raw): 15.95 / 14.68 -> 22.03 / 21.44 | Helped |
| Replace DPO-PL with EPA on UF-all | mistral-7b-sft-beta · MT-Bench | score: 7.57 -> 7.58 | Helped |
| Add on-policy weighting | DPO · Alpaca-Eval 2.0 | win-rate (length controlled / raw): 17.43 / 15.24 -> 21.90 / 21.04 | Helped |
| Add on-policy weighting | EPA · Alpaca-Eval 2.0 | win-rate (length controlled / raw): 19.20 / 19.26 -> 22.80 / 22.26 | Helped |
| Add constant margin | DPO · Alpaca-Eval 2.0 | win-rate (length controlled / raw): 17.43 / 15.24 -> 22.33 / 20.61 | Helped |
| Add constant margin | EPA · Alpaca-Eval 2.0 | win-rate (length controlled / raw): 19.20 / 19.26 -> 23.00 / 22.47 | Helped |
| Use only weak negatives instead of strong negatives (ablation) | EPA · Alpaca-Eval 2.0 (UF-binarized) | win-rate (length controlled / raw): 17.43 / 15.24 -> 9.37 / 6.74 | Hurt |
| Add weak negatives to one strong negative (ablation) | EPA · Alpaca-Eval 2.0 (UF-binarized) | win-rate (length controlled / raw): 17.43 / 15.24 -> 21.14 / 20.55 | Helped |
| Increase weak negatives from 1 to 2 with one strong negative (ablation) | EPA · MT-Bench (UF-binarized) | score: 7.29 -> 7.71 | Helped |
| Increase weak negatives from 2 to 6 with one strong negative (ablation) | EPA · Alpaca-Eval 2.0 (UF-binarized) | win-rate (length controlled / raw): 19.20 / 19.26 -> 16.63 / 15.78 | Hurt |
Surprising findings
- Adding the same weak negatives as extra pairs to DPO gives no advantage over EPA.
- Weak negatives alone perform poorly (1:0:2 gives 9.37 / 6.74), but combined with strong negatives they improve alignment.
Limitations
- EPA is only an empirical approximation of the IPM MLE, so approximation error remains in practice.
- EPA is less computationally and memory efficient than DPO.
- Loss modification tricks designed for DPO do not directly transfer to EPA, and only two were tested.
Experimental setup
Datasets: UltraFeedback UltraFeedback binarized
Benchmarks: AlpacaEval 2.0 MT-Bench GSM8K MMLU Winograd
Metrics: length-controlled win rate win rate MT-Bench score accuracy
Baselines: DPO IPO KTO NCA DPO-PL infoNCA CPO R-DPO SimPO WPO
Builds on: DPO Bradley-Terry model energy discrepancy infoNCA SimPO
Closest papers by meaning
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
- Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking (ICLR 2025 · similarity 0.90)
- As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.89)
- Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.89)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.89)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.