Research map / Large language models / Alignment and preferences
$\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes β-DPO, which dynamically calibrates the DPO β per batch based on reward discrepancy of pairs and filters outlier pairs with a β-guided 3σ criterion, improving DPO across Pythia models, Llama3-8B and Mistral-7B.
Problem
DPO performance is sensitive to the fixed trade-off parameter β and to the quality of preference pairs, and outliers in preference data distort the optimal β choice.
Approach
Adjust β at batch level as a function of the average implicit reward discrepancy (smaller β for low-gap, informative pairs, larger β for high-gap pairs), with a moving-average threshold and a Gaussian-probability-based filter that drops low-probability (outlier) samples from each batch.
Key results
- On Anthropic HH, β-DPO raises Pythia-410M/1.4B/2.8B win rates from 26.19/42.78/51.51 (DPO) to 30.18/48.67/57.07.
- On AlpacaEval 2, β-SimPO improves Llama3-Instruct 8B LC win rate from 44.38 to 46.03 and Mistral-Instruct 7B from 28.50 to 30.48 over SimPO.
- Batch-level calibration outperforms instance-level calibration and vanilla DPO across mixture ratios; at 40% high-gap, vanilla DPO gets 29.15 vs 34.92 for batch-level.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dynamic β calibration and β-guided data filtering | DPO · Anthropic HH, Pythia-410M | win rate: 26.19 -> 30.18 | Helped |
| Add dynamic β calibration and β-guided data filtering | DPO · Anthropic HH, Pythia-1.4B | win rate: 42.78 -> 48.67 | Helped |
| Add dynamic β calibration and β-guided data filtering | DPO · Anthropic HH, Pythia-2.8B | win rate: 51.51 -> 57.07 | Helped |
| Add dynamic β without data filtering (ablation) | DPO · Anthropic HH, Pythia-410M and Pythia-2.8B | win rate: 26.19 -> 27.15; 51.51 -> 55.19 | Helped |
| Add β-guided data filtering without dynamic β (ablation) | DPO · Anthropic HH, Pythia-410M and Pythia-2.8B | win rate: 26.19 -> 29.03; 51.51 -> 53.42 | Helped |
| Use batch-level rather than instance-level β calibration (ablation) | DPO on Anthropic HH mixtures · Anthropic HH mixtures (10%, 20%, 30%, and 40%) | win rate: 49.18−1.97% vs 57.68+14.69%; 49.82−1.46% vs 56.15+11.06%; 44.42−7.36% vs 51.25+6.88%; 16.82−42.30% vs 34.92+19.79% | Helped |
| Apply β-DPO to DPO | DPO with Mistral-Instruct (7B) · AlpacaEval 2 | length-controlled win rate: 20.98 -> 23.56 | Helped |
| Apply dynamic β to SimPO | SimPO with Mistral-Instruct (7B) · AlpacaEval 2 | length-controlled win rate: 28.50 -> 30.48 | Helped |
| Apply β-DPO to DPO | DPO with Llama3-Instruct (8B) · AlpacaEval 2 | raw win rate: 37.38 -> 38.21 | Helped |
| Apply dynamic β to SimPO | SimPO with Llama3-Instruct (8B) · AlpacaEval 2 | raw win rate: 38.97 -> 40.18 | Helped |
| Use a fixed M0 rather than moving-average updating (ablation) | β-DPO · Anthropic HH, Pythia-2.8B | win rate: 53.35; 54.00; 53.85; 55.61; 53.19; 56.75 vs 57.07 | Hurt |
Surprising findings
- Instance-level dynamic β can degrade performance below vanilla DPO (e.g., 16.82 vs 29.15 at 40% mixture ratio), indicating instability from per-sample β.
- On AlpacaEval 2 with Mistral-7B, β-DPO's raw win rate (20.42) is lower than DPO's (21.60) even though its length-controlled win rate is higher.
Limitations
- Evaluated mainly on small Pythia models and single-turn tasks; scalability to models above 7B is unexplored.
- Relies on GPT-4 win rates as the main metric; authors note need for more advanced evaluators.
- Introduces hyperparameter α and sensitivity to ρ and M0 settings that vary across model sizes.
Experimental setup
Datasets: Anthropic HH Reddit TL;DR summarization UltraChat-200k
Benchmarks: Anthropic HH Reddit TL;DR summarization AlpacaEval 2
Metrics: win rate vs chosen (GPT-4 judged) length-controlled win rate raw win rate
Baselines: DPO IPO KTO SPPO SimPO DPO + Dynamic β DPO + Data Filtering
Builds on: DPO SimPO IPO KTO SPPO
Closest papers by meaning
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.92)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.91)
- Provably Robust DPO: Aligning Language Models with Noisy Feedback (ICML 2024 · similarity 0.90)
- Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization (NeurIPS 2025 · similarity 0.90)
- Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.