Research map / Reinforcement learning / RL from feedback
Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that direct alignment algorithms (DPO, IPO, SLiC) exhibit reward over-optimization-style degradation at high KL budgets, fits a scaling law from RLHF to GPT-4 win rates, and explains it via under-constrained objectives and OOD probability mass in a toy MDP.
Problem
Reward over-optimization (hacking) is well characterized for RLHF with a learned proxy reward, but it is unclear how it manifests in direct alignment algorithms (DAAs) that have no explicit reward model.
Approach
Treats DAA win-rate versus KL budget as an over-optimization curve, fits the RLHF reward scaling law R(d)=d(alpha - beta log d) to GPT-4 win rates as a gold proxy, and attributes the degradation to under-constrained (non-strictly-convex) DAA objectives that push probability mass onto out-of-distribution responses.
Key results
- DPO, IPO and SLiC all show hump-shaped win-rate versus KL curves at 1B, 2.8B and 6.9B, with performance degrading at high KL budgets.
- Under lower KL budgets, most runs peak within the first 25% of an epoch and then degrade while KL keeps rising.
- The RLHF scaling law d(alpha - beta log d) halves the RMSE versus a quadratic fit between KL and GPT-4 win rate for DAAs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the KL budget | DPO, IPO, and SLiC · Reddit TL;DR; Pythia 1B, 2.8B, and 6.9B | GPT-4 win rate | Mixed |
| continue training beyond the first 25% of an epoch | DPO, IPO, and SLiC · Reddit TL;DR; Pythia 6.9B | GPT-4 win rate | Hurt |
| use IPO instead of DPO or SLiC | Direct Alignment Algorithms · Reddit TL;DR; Pythia models | over-optimization susceptibility | Helped |
| increase model size from Pythia 1B to Pythia 6.9B | DPO, IPO, and SLiC · Reddit TL;DR | GPT-4 win rate | Helped |
| use length regularization | DPO · Reddit TL;DR; Pythia 2.8B | length-corrected GPT-4 win rate | Helped |
| use length regularization | DPO · Reddit TL;DR; Pythia 2.8B, lower KL constraint region | GPT-4 win rate | Hurt |
| use length regularization | DPO · Reddit TL;DR; Pythia 2.8B | reward over-optimization | No clear effect |
| fit the reward-versus-KL scaling law against square-root KL instead of using a quadratic fit against KL | Direct Alignment Algorithms · Reddit TL;DR; GPT-4 win rates | RMSE: halves the RMSE | Helped |
| increase model capacity | DPO · Reddit TL;DR; Pythia models | length extrapolation (R²) | Helped |
| use a smaller KL budget | DPO · Reddit TL;DR; Pythia models | length extrapolation (R²) | Hurt |
| increase the DAA implicit reward classification accuracy | DPO and SLiC · Reddit TL;DR; Pythia models | GPT-4 win rate | No clear effect |
| increase the DAA implicit reward classification accuracy | IPO · Reddit TL;DR; within each Pythia model size | GPT-4 win rate | No clear effect |
Surprising findings
- Implicit DAA reward accuracy and DAA loss show little to no correlation with policy win rate within a model size.
- Decreasing likelihoods of preferred responses during DPO training appear necessary for improvement rather than a sign of failure.
Limitations
- Experiments limited to model sizes up to 6.9B; larger scales not tested due to compute.
- Gold reward proxied by GPT-4 win rates, not human judgments.
- Theoretical OOD explanation relies on toy MDP and a proposition that is only sketched.
Experimental setup
Datasets: OpenAI TL;DR summarization preferences (92K) Anthropic Helpfulness-Harmlessness
Benchmarks: OpenAI TL;DR held-out prompts Anthropic-HH
Metrics: GPT-4 win rate KL divergence implicit reward accuracy RMSE of scaling law fit R^2 of length regression
Baselines: DPO IPO SLiC-HF
Builds on: DPO IPO SLiC-HF Scaling Laws for Reward Model Overoptimization
Closest papers by meaning
- Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling (NeurIPS 2025 · similarity 0.93)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.91)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
- AlphaPO: Reward Shape Matters for LLM Alignment (ICML 2025 · similarity 0.91)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
- Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.