Research map / Reinforcement learning / RL from feedback

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization

ICML 2025 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AlphaDPO, a DPO variant that replaces the static reference with an implicit reference U(y|x)(π_θ/π_ref)^α, yielding an instance-adaptive reward margin, and reports SOTA AlpacaEval 2 LC and Arena-Hard results on Llama3, Mistral and Gemma2 instruct models.

Problem

Offline preference optimization for LLM alignment: DPO's static reference model can fail to separate preferred and rejected responses, while SimPO's fixed target margin ignores instance-wise preference strength.

Approach

Redefine the reference distribution as an implicit reference proportional to a uniform distribution times the policy-to-reference ratio raised to α, which turns the DPO margin into a constant offset plus an instance-specific term M (a log-ratio difference), with stop-gradient and z-score normalization.

Key results

• AlphaDPO reaches 58.7% AlpacaEval 2 LC win rate on Llama3-Instruct v0.2 (8B), versus 55.6% for SimPO.
• On Arena-Hard, AlphaDPO achieves 35.7% WR on Llama3-Instruct v0.2 and 60.8% WR on Gemma2-Instruct (9B).
• Ablation: removing stop-gradient collapses Llama3 v0.2 LC win rate to 2.7%, while removing z-score normalization drops Arena-Hard LC from 36.3 to 28.4.

Measured effects reported in the paper

Surprising findings

• Removing the stop-gradient on the implicit reference causes a catastrophic drop (AlpacaEval 2 LC 2.7% on Llama3-Instruct v0.2).
• The best α differs across benchmarks: style-controlled Arena-Hard peaks at intermediate α while AlpacaEval 2 LC increases monotonically with α.

Limitations

• Introduces an extra hyperparameter α that requires manual tuning, and optimal α differs across benchmarks.
• Remains an offline method; the authors note the online extension is future work.
• Theoretical margin equivalence relies on approximating sequential KL by full-sequence log-ratio and on a uniform-reference assumption.

Experimental setup

Datasets: princeton-nlp/llama3-ultrafeedback-armorm princeton-nlp/mistral-instruct-ultrafeedback princeton-nlp/gemma2-ultrafeedback-armorm

Benchmarks: AlpacaEval 2 Arena-Hard

Metrics: length-controlled win rate raw win rate style-controlled win rate

Baselines: SFT DPO IPO CPO KTO ORPO R-DPO SimPO TDPO

Builds on: DPO SimPO TDPO R-DPO weak-to-strong preference optimization

Closest papers by meaning

• Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.93)
• Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.92)
• MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment (ICML 2025 · similarity 0.91)
• Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
• Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.91)
• Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.91)