Research map / Reinforcement learning / RL from feedback

Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IS-DAAs, which reweight offline DAA losses (DPO/IPO) with a clipped sequence-level importance ratio so the regularization approximates on-policy reverse KL, mitigating reward over-optimization without online sampling.

Problem

Direct alignment algorithms trained on offline preference data drift from the reference policy and suffer reward over-optimization and early convergence, as their implicit KL regularization is only a local approximation around the offline distribution.

Approach

Multiply the offline DAA objective by a truncated (clipped at epsilon) sequence-level importance ratio between the policy and reference, so the expected weighted squared-loss gradient matches the reverse-KL gradient under the policy, without on-policy sampling.

Key results

• On TL;DR summarization at low regularization (beta=0.01), IS-DPO avoids over-optimization while maintaining a significantly lower KL budget than baselines, where other methods except chi-PO degrade.
• On Anthropic-HH, IS-DPO reaches 84.60% reward-model win rate and 74.0% GPT-4 win rate, versus 81.77% and 71.1% for DPO.
• On MT-Bench IS-DPO achieves GPT-4 score 5.44 vs 5.39 for DPO; on AlpacaEval 2.0 it gets 6.33% vs chi-PO's 6.85%.

Measured effects reported in the paper

Surprising findings

• Length regularization and RPO fail to prevent over-optimization in the low-beta summarization setting.
• IS-DPO's penalty acts like an implicit early stopping at a specific KL level without requiring an evaluation set.

Limitations

• Experiments limited to a 3B model due to computational resources; larger models not tested.
• Assumes Bradley-Terry model of human preferences.
• Gains on AlpacaEval 2.0 are not uniform, with chi-PO scoring higher.

Experimental setup

Datasets: Reddit TL;DR summarization Anthropic Helpful and Harmless (HH) UltraFeedback

Benchmarks: Reddit TL;DR summarization win rate Anthropic-HH AlpacaEval 2.0 MT-Bench

Metrics: win rate KL divergence GPT-4 win rate GPT-4 score

Baselines: DPO IPO RPO chi-PO Length-DPO Online-DPO

Builds on: Direct Preference Optimization Identity Preference Optimization Truncated importance sampling Regularized Preference Optimization

Closest papers by meaning

• Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.93)
• AlphaPO: Reward Shape Matters for LLM Alignment (ICML 2025 · similarity 0.91)
• Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.90)
• Bootstrapping Language Models with DPO Implicit Rewards (ICLR 2025 · similarity 0.89)
• As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.89)
• Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.89)