Research map / Large language models / Alignment and preferences

AlphaPO: Reward Shape Matters for LLM Alignment

ICML 2025 · Large language models › Alignment and preferences · approach: Active and token-level preference learning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AlphaPO, a direct alignment algorithm whose length-normalized reward uses an alpha-divergence-derived shape controlled by a single parameter alpha (alpha=0 recovers SimPO's log reward), improving AlpacaEval 2 over SimPO and DPO on instruct models.

Problem

Direct alignment algorithms like DPO and SimPO suffer from likelihood displacement and reward over-optimization, and their tuning focuses on learning rate, beta and gamma while ignoring the shape of the reward function.

Approach

Replace SimPO's log reward with a length-normalized alpha-divergence reward r_alpha = beta(1 - pi^(-alpha/|y|))/alpha, so that alpha tunes the reward shape and the strength of likelihood displacement; alpha=0 recovers SimPO.

Key results

• On PairRM data, AlphaPO reaches 33.03% LC on AlpacaEval 2 for Mistral-7B-Instruct vs 29.71 for SimPO and 21.96 for DPO.
• On PairRM data, AlphaPO reaches 45.37% LC for Llama-3-8B-Instruct vs 42.05 for SimPO and 39.18 for DPO.
• With SPPO applied on top, AlphaPO improves LC from 45.37 to 47.42, versus SimPO's 42.05 to 45.06.

Measured effects reported in the paper

Surprising findings

• On Gemma-2-9B-Instruct with ArmoRM data, AlphaPO has a higher LC but slightly lower raw win rate and Arena-Hard score than SimPO and DPO.
• f-DPO-style alpha values do not improve generalization in DPO, whereas non-zero alpha helps when combined with length normalization.

Limitations

• Gains on Gemma-2-9B with ArmoRM data are mixed, with lower raw win rate and Arena-Hard scores.
• Gradient and dynamics analysis is for a single example or gradient flow; the multi-example interaction is only studied empirically.
• Alpha and other hyperparameters are tuned per model on the evaluation benchmark (AlpacaEval 2), which may bias reported results.

Experimental setup

Datasets: UltraFeedback (binarized) HellaSwag TruthfulQA

Benchmarks: AlpacaEval 2.0 Arena-Hard HellaSwag TruthfulQA

Metrics: length-controlled win rate win rate accuracy standard deviation of win rate

Baselines: DPO SimPO SPPO f-DPO

Builds on: SimPO DPO f-DPO ORPO SPPO

Closest papers by meaning

• Bootstrapping Language Models with DPO Implicit Rewards (ICLR 2025 · similarity 0.93)
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML 2024 · similarity 0.91)
• Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.91)
• Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling (NeurIPS 2025 · similarity 0.91)
• Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.90)
• $f$-GRPO & Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment (NeurIPS 2026 · similarity 0.90)