Research map / Reinforcement learning / RL from feedback
Confronting Reward Model Overoptimization with Constrained RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies reward model overoptimization in composite RMs (METEOR and intent) on DailyDialog with GPT-2, and proposes constrained RLHF (Lagrangian PPO variants) with proxy-point thresholds, plus a Nelder-Mead single-run threshold search.
Problem
Composite reward models combining several imperfect RMs are vulnerable to overoptimization, and correlations between component RMs shift the points where each stops being a useful proxy; weighting them manually is hard and early stopping is problematic.
Approach
Treat each component RM's overoptimization point (proxy point) as a constraint threshold in a constrained MDP, and learn the RM weights as Lagrange multipliers via primal-dual PPO; proxy points are found from a fitted surface over multiple runs or by Nelder-Mead search within a single run.
Key results
- xi-PPO and mu-PPO achieve the best evaluation performance over training, with xi-PPO slightly better at the end of training (5 seeds).
- Proxy-point thresholds outperform thresholds 10% lower or higher, and correlation-aware proxy points outperform independently estimated ones.
- NM-PPO obtains strong performance within a single run, saving computation relative to multi-run PPO and xi-PPO hyperparameter search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dynamic Lagrange multipliers with inequality constraints (μ-PPO) | PPO on GPT-2 · DailyDialog | evaluation score | Helped |
| Use dynamic Lagrange multipliers with equality constraints (ξ-PPO) | PPO on GPT-2 · DailyDialog | evaluation score | Helped |
| Use constrained training rather than fixed-weight PPO | GPT-2 dialogue generation · DailyDialog | stability over longer training | Helped |
| Use All-PPO constraints on component rewards | PPO on GPT-2 · DailyDialog | evaluation score | Hurt |
| Set thresholds 10% below the proxy points (ablation) | ξ-PPO · DailyDialog | evaluation score | Hurt |
| Set thresholds 10% above the proxy points (ablation) | ξ-PPO · DailyDialog | evaluation score | Hurt |
| Use proxy points accounting for reward-model correlation rather than independently estimated points (ablation) | ξ-PPO · DailyDialog | evaluation score | Helped |
| Search thresholds with Nelder-Mead during a single run | ξ-PPO · DailyDialog | computation | Helped |
| Continue optimizing intent reward after reaching its proxy point | μ-PPO · DailyDialog | final evaluation score | Hurt |
Surprising findings
- Fixing Lagrange multipliers at their optimal values is not guaranteed to converge to a saddle point, and All-PPO performed worse than other constrained methods due to instability.
Limitations
- Experiments limited to a single dialogue setting with two component RMs and a small feasible region.
- Requires some access to ground-truth evaluation queries to identify proxy points.
- Gradient descent-ascent only guarantees convergence of averaged iterates, not the final policy.
Experimental setup
Datasets: DailyDialog
Benchmarks: DailyDialog
Metrics: averaged normalized lexical and diversity evaluation score ( METEOR and intent reward satisfaction
Baselines: PPO PPO-SAT All-PPO xi-PPO fixed thresholds
Builds on: PPO constrained MDP RL4LMs Nelder-Mead
Closest papers by meaning
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.91)
- Reward Model Ensembles Help Mitigate Overoptimization (ICLR 2024 · similarity 0.91)
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.91)
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.90)
- Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation (NeurIPS 2024 · similarity 0.90)
- Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.