Research map / Reinforcement learning / RL from feedback
Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Iterative Data Smoothing (IDS), which replaces hard preference labels with model-predicted soft labels during reward model training, to mitigate reward overfitting and overoptimization in RLHF.
Problem
Reward models in RLHF overfit after about one epoch (test cross-entropy rises), and policy optimization against them causes reward overoptimization, especially with long-tailed preference data coverage.
Approach
During each training epoch, update the reward model on the current labels and then update the labels toward the model's predicted preference probabilities (a weighted average with step size beta), so rarely compared pairs keep near-initial labels and are implicitly penalized, similar to pessimism.
Key results
- In the bandit setting, MLE and pessimistic MLE overfit and overoptimize while IDS's test cross-entropy keeps decreasing until convergence and recovers the ground-truth reward at large KL.
- On HH with Pythia reward models, MLE begins to overfit after 1-2 epochs while IDS loss decreases stably until convergence.
- In PPO policy learning on HH, MLE suffers reward overoptimization within a few thousand steps while ground-truth reward keeps growing under IDS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iterative model-based soft-label updates instead of vanilla MLE | Reward learning in multi-armed bandits · Multi-armed bandit | population cross-entropy loss | Helped |
| Use iterative model-based soft-label updates instead of pessimistic MLE | Reward learning in multi-armed bandits · Multi-armed bandit | population cross-entropy loss | Helped |
| Use iterative model-based soft-label updates instead of vanilla MLE | Policy learning from learned rewards · Multi-armed bandit | ground-truth reward | Helped |
| Use iterative model-based soft-label updates instead of MLE | Reward learning with neural reward models · HH | evaluation loss | Helped |
| Use iterative model-based soft-label updates instead of MLE | Policy learning with neural reward models · HH | ground-truth reward | Helped |
| Use iterative model-based soft-label updates instead of MLE | Reward learning · TLDR | test accuracy | No clear effect |
| Use iterative model-based soft-label updates instead of MLE | Policy learning · TLDR | policy reward | Helped |
| Replace the standard IDS soft-label update with confidence-weighted original-label loss (IDS V2) (ablation) | IDS · Neural network reward learning | practical performance | Hurt |
Surprising findings
- Pessimistic MLE also suffers from both overfitting and overoptimization in the unbounded-reward bandit case.
- An alternative IDS V2 with confidence weighting has better asymptotic guarantees but performs worse in practice and also overfits.
Limitations
- Theoretical analysis is limited to tabular bandit setting and heuristic two-timescale argument; no formal guarantee for neural networks.
- Only one random seed in bandit experiments.
- Possible bias in human preference data is not addressed; IDS may not fix bias-driven issues.
Experimental setup
Datasets: Anthropic HH (Dahoas/static-hh) TLDR (CarperAI/openai_summarize_comparisons) simulated bandit data
Benchmarks: Anthropic HH TLDR simulated 3-armed and K-armed bandit
Metrics: population/test cross-entropy loss test accuracy ground-truth reward vs KL divergence suboptimality
Baselines: MLE (vanilla cross-entropy reward training) pessimistic MLE
Builds on: Bradley-Terry-Luce model pessimistic MLE proximal policy optimization knowledge distillation
Closest papers by meaning
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.90)
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.90)
- Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective (ICML 2025 · similarity 0.89)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.89)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.89)
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.