Research map / Reinforcement learning / RL from feedback
Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a lightweight uncertainty estimate from reward-model last-layer embeddings and uses it in ADVPO, a distributionally robust (max-min) policy optimization that mitigates reward overoptimization in RLHF.
Problem
Reward overoptimization in RLHF, where the policy exploits an imperfect proxy reward model; existing uncertainty-based fixes need reward ensembles that are costly in memory and compute.
Approach
Derive a reward confidence interval from last-layer embeddings via neural bandit theory (inverse covariance of embeddings over preference data), then optimize the policy against the worst-case projection weight inside that confidence ball (ADVPO), with a closed-form inner minimization and reference responses to avoid excess pessimism.
Key results
- On Anthropic HH, ADVPO beats PPO with 31.0% win / 20.0% lose (delta +11.0), and on TL;DR with 75.0% win / 18.0% lose (delta +57.0).
- The lightweight CI uncertainty achieves Pearson correlation 0.984 (Anthropic HH) and 0.994 (TL;DR) with reward differences, versus 0.787 and -0.594 for a 3B ensemble (ENS-3B).
- ADVPO with a single 7B reward model outperforms the three-7B-model ensemble ENS-s-7B on TL;DR (delta +57.0 vs +27.0 against PPO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ADVPO instead of PPO | Llama7B RLHF policy · Anthropic HH | human-assisted win rate: 31.0 win rate; ↑11.0 | Helped |
| Use ADVPO instead of PPO | Llama7B RLHF policy · TL;DR | human-assisted win rate: 75.0 win rate; ↑57.0 | Helped |
| Use last-layer embedding CI uncertainty instead of ENS-3B uncertainty | reward uncertainty estimation · Anthropic HH | Pearson correlation with reward difference: 0.787 -> 0.984 | Helped |
| Use last-layer embedding CI uncertainty instead of ENS-3B uncertainty | reward uncertainty estimation · TL;DR | Pearson correlation with reward difference: -0.594 -> 0.994 | Helped |
| Add reference responses to ADVPO (ablation) | ADVPO · Anthropic HH | human-assisted win rate against PPO: 36.5 -> 31.0 | Mixed |
| Add reference responses to ADVPO (ablation) | ADVPO · TL;DR | human-assisted win rate against PPO: 74.0 -> 75.0 | Helped |
| Use ADVPO instead of LoraEns | Llama7B RLHF policy · Anthropic HH | human-assisted win rate: 65.5 win rate; ↑46.5 | Helped |
| Use ADVPO instead of LoraEns | Llama7B RLHF policy · TL;DR | human-assisted win rate: 84.0 win rate; ↑68.0 | Helped |
| Increase the ensemble count from three to five 3B reward models (ablation) | ENS-3B uncertainty estimation · TL;DR | alignment of uncertainty with reward difference | No clear effect |
| Use 3B policy and reward models for CI uncertainty estimation (ablation) | lightweight CI uncertainty estimation · Anthropic HH and TL;DR | uncertainty tracking reward divergence | Helped |
Surprising findings
- Ensembles of three 3B reward models gave uncertainty that was poorly or negatively correlated with reward divergence on TL;DR, even with five ensembles.
- Reward scaling in PPO significantly hindered learning in this setting, so the authors removed it.
Limitations
- Uncertainty is computed only from the last layer of the reward model.
- Uncertainty is used only for training, not for active data selection.
- Experiments not run at larger scale (e.g., 70B).
Experimental setup
Datasets: Anthropic HH TL;DR
Benchmarks: Anthropic HH TL;DR
Metrics: gold reward proxy reward KL divergence win rate lose rate tie rate Pearson correlation between uncertainty and reward differenc
Baselines: PPO PPO-ref ENS-s (UWO) ENS-ref ENS-s-7B LoraEns LWUN-s ADVPO-noRef
Builds on: PPO uncertainty-weighted optimization LoRA ensembles for reward uncertainty neural bandits uncertainty quantification
Closest papers by meaning
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.92)
- Reward Model Ensembles Help Mitigate Overoptimization (ICLR 2024 · similarity 0.90)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.90)
- Confronting Reward Model Overoptimization with Constrained RLHF (ICLR 2024 · similarity 0.90)
- On the Robustness of Reward Models for Language Model Alignment (ICML 2025 · similarity 0.89)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.