Research map / Reinforcement learning / RL from feedback
Reward Model Ensembles Help Mitigate Overoptimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether ensemble-based conservative objectives (worst-case and uncertainty-weighted optimization) mitigate reward model overoptimization for best-of-n sampling and PPO in a synthetic AlpacaFarm setup with 25% label noise.
Problem
Learned proxy reward models in RLHF are susceptible to overoptimization, where optimizing the proxy reward degrades the true (gold) reward; this is studied in a synthetic setup with label noise to better mirror real-world conditions.
Approach
Apply ensembles of proxy reward models with conservative policy objectives (worst-case min over members and mean-minus-variance UWO) instead of a single reward model, evaluated under BoN and PPO, with and without 25% label noise.
Key results
- For best-of-n sampling, conservative ensemble optimization eliminates overoptimization and improves final performance by up to ~30% (no noise) and ~75% (25% label noise) over single reward models with 44M proxies.
- For PPO, worst-case and uncertainty-weighted optimization reduce overoptimization and outperform single reward models; adding a KL penalty of 0.01 prevents overoptimization without performance cost.
- Ensemble gains are reported as orthogonal to scaling the proxy reward model size, with UWO still giving gains with a 1.3B proxy reward model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WCO or UWO instead of a single reward model | PPO policy optimization · AlpacaFarm, no label noise and 25% label noise | gold reward model performance / overoptimization | Helped |
| Use WCO or UWO instead of a single reward model | BoN policy optimization · AlpacaFarm, no label noise and 25% label noise | gold reward model performance: up to ∼ 30%; up to ∼ 75% | Helped |
| Use mean reward across an ensemble | BoN policy optimization · AlpacaFarm, 25% label noise | overoptimization | Hurt |
| Add a KL penalty of 0.01 to WCO or UWO | PPO policy optimization · AlpacaFarm | overoptimization: 0.01 | Helped |
| Use a KL penalty alone to prevent overoptimization | PPO with a single reward model · AlpacaFarm, 25% label noise | gold reward model performance: 0.2 | Hurt |
| Increase reward-model size (ablation) | Ensemble-based policy optimization · BoN and PPO, 25% label noise | gold reward model performance | Helped |
| Increase reward-model training dataset size (ablation) | Ensemble-based policy optimization · BoN and PPO, 25% label noise | gold reward model performance | Helped |
| Increase ensemble size from 3 to 4 members (ablation) | Mean, WCO, and UWO policy optimization · BoN and PPO, 25% label noise | final gold reward model performance | Helped |
| Increase ensemble size from 4 to 5 members (ablation) | Mean, WCO, and UWO policy optimization · BoN and PPO, 25% label noise | final gold reward model performance | No clear effect |
| Vary the UWO uncertainty penalty weight (ablation) | UWO policy optimization · BoN and PPO | final gold reward model performance | No clear effect |
| Increase label noise to 35% (ablation) | WCO and UWO · BoN and PPO, 44M reward models | gold reward model performance / overoptimization: 35% | Helped |
| Extend PPO optimization to 10k steps (ablation) | Ensemble methods · PPO, 44M reward models, 25% label noise | gold reward model performance stability: 10k | No clear effect |
Surprising findings
- Mean optimization, despite being an ensemble, overoptimizes under 25% label noise for best-of-n sampling.
- Reward model validation loss is not predictive of robustness to overoptimization.
Limitations
- Experiments limited to small Pythia models and a single dataset (AlpacaFarm), with offline RLHF only.
- BoN evaluation limited to n=12,500 samples due to compute, so high-KL regime is not explored.
- WCO can incur a performance penalty due to its conservative nature; PPO does not fully eliminate overoptimization without a KL penalty.
Experimental setup
Datasets: AlpacaFarm Alpaca
Benchmarks: AlpacaFarm
Metrics: gold reward model score KL divergence final policy win-rate
Baselines: single reward model optimization mean optimization
Builds on: Scaling laws for reward model overoptimization (Gao et al., 2023) proximal policy optimization AlpacaFarm
Closest papers by meaning
- Confronting Reward Model Overoptimization with Constrained RLHF (ICLR 2024 · similarity 0.91)
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.91)
- Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation (NeurIPS 2024 · similarity 0.90)
- Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer (NeurIPS 2024 · similarity 0.89)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.89)
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.