Research map / Reinforcement learning / RL from feedback

Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking

ICLR 2025 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies two types of reward hacking in offline preference optimization from data sparsity and proposes POWER-DL, combining weighted-entropy robust rewards with dynamic preference labels, which outperforms DPO and other PO variants on AlpacaEval 2.0 and Arena-Hard.

Problem

Reward hacking in offline preference optimization: statistical fluctuations in sparsely covered preference data cause learned policies to overweight subpar responses (Type I) or underweight good responses and degrade the initial model (Type II).

Approach

Combine a robust (adversarial) reward maximization with weighted-entropy regularization (POWER), giving a single-step objective with length-normalized SFT term, plus dynamic preference labels that shrink gradients on poorly covered pairs so the policy stays near the initial model there.

Key results

• POWER-DL on Helpsteer2 Llama3-8B-Base reaches 31.52 LC and 31.44 WR on AlpacaEval 2.0 and 21.5 on Arena-Hard, versus 18.52 / 14.99 / 10.0 for DPO.
• POWER-DL on Zephyr Llama3-8B-Base gets 27.00 LC AlpacaEval and 17.3 Arena-Hard, versus 22.53 and 13.3 for DPO.
• Paper reports POWER-DL improves over DPO by up to 13.0 points on AlpacaEval 2.0 and 11.5 on Arena-Hard, and up to 7.0 points on GSM8K.

Measured effects reported in the paper

Surprising findings

• Divergence-minimization methods (DPO, IPO, chiPO) can fail even when the learned policy stays close to the initial model, because small divergence does not induce enough pessimism under sparse coverage.
• SimPO can significantly degrade GSM8K performance of the initial model in some settings.

Limitations

• Theoretical analysis of dynamic labels is restricted to a bandit/softmax setting; unified analysis with POWER and general function approximation is left open.
• Reward hacking analysis focuses on statistical fluctuations from partial coverage and does not address reward misspecification.

Experimental setup

Datasets: OpenAssistant2 (oasst2) HelpSteer2 UltraChat-200K UltraFeedback

Benchmarks: AlpacaEval 2.0 Arena-Hard MT-Bench MMLU MMLU-PRO ARC-Challenge HellaSwag Winogrande TruthfulQA IFEval GSM8K

Metrics: length-controlled win rate win rate accuracy

Baselines: DPO IPO SPPO chiPO cDPO R-DPO DPO+SFT CPO SLiC-HF RRHF ORPO SimPO

Builds on: DPO SimPO IPO chiPO DPO+SFT weighted entropy

Closest papers by meaning

• AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.91)
• Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.90)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
• Rectifying Shortcut Behaviors in Preference-based Reward Learning (NeurIPS 2025 · similarity 0.90)
• Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.90)
• Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model (ICML 2025 · similarity 0.90)