Research map / Reinforcement learning / RL from feedback
KL Penalty Control via Perturbation for Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ε-DPO, which adaptively sets the DPO KL penalty β per preference pair by checking whether perturbing β (via reused policy and reference logits) increases preference confidence.
Problem
DPO uses a static KL penalty β for all preference pairs, which may be suboptimal; existing dynamic-KL methods either operate at batch level (β-DPO) or periodically reset the reference policy (TR-DPO).
Approach
Estimate policies at perturbed temperatures β/(1±ε) by linearly mixing the current policy's and reference policy's logits, then per instance raise or lower β depending on whether the log-likelihood-ratio logit increases or decreases monotonically under the perturbation; β is then averaged over the batch for the next step.
Key results
- On Llama-3-Instruct (8B), ε-DPO reaches 46.4% LC AlpacaEval 2 and 36.7% Arena-Hard, above DPO (40.3 / 32.6) and SimPO (44.7 / 33.8).
- On Mistral-Instruct (7B), ε-DPO reaches 35.6% LC AlpacaEval 2 vs DPO 26.8% and SimPO 32.1%, but lower Arena-Hard (17.2 vs CPO 22.6).
- On Qwen2.5-7B-Instruct with Llama-3 hyperparameters, ε-DPO achieves 67.5 Arena-Hard and 9.1 MT-Bench, above DPO (66.8, 8.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| instance-level adaptive KL penalty control (ε-DPO) replacing static β in DPO | DPO on Mistral-Instruct (7B) · UltraFeedback AlpacaEval 2 / Arena-Hard / MT-Bench | AlpacaEval 2 LC and Arena-Hard WR: AlpacaEval 2 LC 26.8 to 35.6; Arena-Hard 16.3 to 17.2 | Helped |
| instance-level adaptive KL penalty control (ε-DPO) replacing static β in DPO | DPO on Llama-3-Instruct (8B) · UltraFeedback AlpacaEval 2 / Arena-Hard / MT-Bench | AlpacaEval 2 LC, Arena-Hard WR, MT-Bench: LC 40.3 to 46.4; Arena-Hard 32.6 to 36.7; MT-Bench 8.0 unchanged | Helped |
| instance-level adaptive KL penalty control (ε-DPO) replacing static β in DPO | DPO on Qwen2.5-7B-Instruct using Llama-3 hyperparameters · UltraFeedback AlpacaEval 2 LC, MT-Bench | AlpacaEval 2 LC and MT-Bench: LC 41.6 to 42.5; MT-Bench 8.9 to 9.1; WR 46.3 to 46.1 (no gain) | Helped |
| instance-level adaptive KL penalty control (ε-DPO) | DPO on Mistral-Instruct (7B) · UltraFeedback AlpacaEval 2 WR | win rate (%): WR 29.6 vs SimPO 34.8 (below best baseline) | Hurt |
| instance-level adaptive KL penalty control (ε-DPO) | DPO on Llama-3-Instruct (8B) · Open LLM Leaderboard (6 tasks) | average score: average 64.30 vs DPO 65.86; GSM8K drops 49.81 to 40.26 | Hurt |
| instance-level adaptive KL penalty control (ε-DPO) | DPO on Llama-3-Instruct (8B) · AlpacaEval 2 WR vs TR-DPO | AlpacaEval 2 WR (%): WR 44.9 vs TR-DPO^τ 47.2 and TR-DPO^α 46.8 | Hurt |
| adaptive instance-level control (ε-DPO) vs batch-level β-DPO and periodic-reference TR-DPO | DPO on Llama-3-Instruct (8B) · AlpacaEval 2 LC | LC (%): ε-DPO 46.4 vs β-DPO 43.4, TR-DPO^α 43.5, TR-DPO^τ 42.8 | Helped |
| using a larger monotonicity-check ε (ε_c=0.02) with ε_s=0.01 instead of shared ε=0.01 (ablation) | ε-DPO on Anthropic-HH · Anthropic-HH win rate vs chosen | win rate (%): 74.2 vs 79.2 default | Hurt |
| smaller monotonicity-check ε (ε_c=0.005) with ε_s=0.01 (ablation) | ε-DPO on Anthropic-HH · Anthropic-HH win rate vs chosen | win rate (%): 76.7 at ε_c=0.005 (ε_s=0.01) vs 79.2 default | Hurt |
| per-instance adaptive β control direction (assign higher β to low-confidence pairs) vs β-DPO's margin-based rule | ε-DPO on Anthropic-HH · qualitative analysis of implicit reward margin | implicit reward margin / logit monotonicity | No clear effect |
| adding the instance-level adaptive KL control (ε-DPO) to DPO | Anthropic-HH KL trade-off · forward KL vs win rate Pareto frontier | win rate at given KL: ε-DPO better than DPO and achieves better KL trade-off than TR-DPO, which induces more KL | Helped |
Surprising findings
- Preference pairs with monotonically increasing logits tend to show low preference confidence, so ε-DPO raises β for them, opposite to β-DPO's decisions based on implicit reward margin.
- SimPO underperforms DPO on Qwen2.5-7B-Instruct despite being competitive on Llama-3-Instruct, whereas ε-DPO still beats DPO.
Limitations
- Requires a reference policy and its logits, adding memory and computation compared with reference-free direct alignment methods.
- The monotonicity criterion assumes the current policy approximates the optimal policy and that monotonicity holds across the whole ε-neighborhood; both can fail early in training, as the upper-bound analysis shows.
- Results on Mistral-Instruct Arena-Hard are weaker than several baselines, and the Open LLM Leaderboard shows a GSM8K drop with ε-DPO for Llama-3 (40.26 vs DPO 49.81).
Experimental setup
Datasets: UltraFeedback Anthropic-HH mistral-instruct-ultrafeedback llama3-ultrafeedback
Benchmarks: AlpacaEval 2 Arena-Hard MT-Bench Anthropic-HH win rate vs chosen (PairRM) Huggingface Open LLM Leaderboard
Metrics: length-controlled win rate win rate score (1-10) forward KL divergence accuracy
Baselines: DPO RRHF SLiC-HF IPO CPO KTO ORPO R-DPO SimPO β-DPO TR-DPO
Builds on: DPO SimPO β-DPO TR-DPO Logit-mixing importance sampling for sampling distribution (reweighting with ref
Closest papers by meaning
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.91)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.91)
- Token-level Direct Preference Optimization (ICML 2024 · similarity 0.91)
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.91)
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.