Research map / Reinforcement learning / RL from feedback
Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BSPO, which models the reward model's in-distribution region via a next-token behavior policy and applies a behavior-supported Bellman operator to penalize OOD values, reducing reward over-optimization in synthetic RLHF experiments.
Problem
Reward over-optimization in RLHF: the policy generates out-of-distribution responses on which the reward model extrapolates poorly, causing proxy reward to diverge from gold reward.
Approach
Define a behavior policy as the next-token distribution of the reward training data to delimit the reward model's in-distribution region, and use a behavior-supported Bellman operator that sets OOD action values to a minimum while leaving ID values unchanged; implemented via a ScoreLM that jointly predicts reward and next-token distribution, combined with PPO.
Key results
- BSPO achieves the highest gold reward across three proxy scales (774M, 1.1B, 2.7B) while most baselines over-optimize.
- Count of behavior-unsupported tokens per response stays low under BSPO during RL, while it rises sharply for baselines.
- With the 2.7B proxy, BSPO attains higher win rate against the initial Alpaca-7B than baselines; BSPO avoids over-optimization at larger KL distances where KL-penalty fails.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate responses containing behavior-unsupported actions rather than only supported actions | Proxy reward model preference prediction · Supported vs. unsupported response comparison pairs | preference prediction accuracy: 75.91\% -> 58.10\% | Hurt |
| Use BSPO value regularization instead of baseline algorithms | RLHF with Alpaca-7B actor · Synthetic setup across three proxy model scales | gold reward | Helped |
| Use behavior-supported value regularization during policy optimization | BSPO · RL response generation | behavior-unsupported actions | Helped |
| Use BSPO instead of KL penalty regularization | RLHF · Larger KL divergence distances | gold reward | Helped |
| Integrate reward prediction and behavior-distribution prediction in one ScoreLM model | ScoreLM · Reward prediction accuracy | prediction accuracy | No clear effect |
Surprising findings
- Proxy reward predictions on unsupported response pairs drop to 58.10% accuracy versus 75.91% on supported pairs, supporting the OOD indicator.
Limitations
- Evaluation uses a synthetic gold-model setup rather than real human preferences.
- Assumes a well-trained reward model; does not address OOD prompts or reward model errors from training.
- Only single-objective reward; multi-reward extension left to future work.
Experimental setup
Datasets: UltraFeedback (binarized)
Benchmarks: synthetic gold-reward setup on UltraFeedback test set
Metrics: gold reward proxy reward win rate vs initial SFT model count of behavior-unsupported tokens
Baselines: Standard PPO KL-Penalty CPPO ENS-UWO ENS-WCO
Builds on: PPO KL penalty Constrained PPO ensemble reward models (UWO/WCO)
Closest papers by meaning
- Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation (NeurIPS 2024 · similarity 0.92)
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.92)
- Confronting Reward Model Overoptimization with Constrained RLHF (ICLR 2024 · similarity 0.91)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.91)
- Reward Model Ensembles Help Mitigate Overoptimization (ICLR 2024 · similarity 0.91)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.