Research map / Reinforcement learning / RL from feedback
Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines reward hacking via correlation between proxy and true reward under a reference policy, proves chi^2 occupancy-measure regularization lower-bounds true-reward improvement, and introduces ORPO, which outperforms action-distribution KL regularization across five environments including RLHF.
Problem
Reward hacking, where optimizing a misspecified proxy reward lowers the true reward, lacks a formal definition and principled mitigation; standard RLHF KL penalties on action distributions lack guarantees.
Approach
Define a proxy as correlated with the true reward under a reference policy's state-action distribution, define reward hacking as true reward dropping below the reference policy, and show that regularizing the chi-squared divergence between occupancy measures (not action distributions or KL) provably lower-bounds true reward improvement; implement via a discriminator-based PPO method (ORPO).
Key results
- Chi-squared OM regularization achieved true reward of -1.15e3 vs -2.28 reference... (traffic) with state-action chi^2 best in glucose (-47.6e3 vs -72.6e3 reference)
- In glucose and pandemic environments, action-distribution regularization fails to improve on the reference policy's true reward, while OM regularization does
- In RLHF, chi^2 action-distribution regularization reached 16.94 true reward vs 16.37 for the reference and 16.81 for KL
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use state-action occupancy measure regularization instead of action distribution regularization | Proxy-reward policy optimization · Traffic control | true reward: -1.29 $\pm$ 0.10 -> -\textbf{1.15} $\pm$ 0.05 | Helped |
| Use state-action occupancy measure regularization instead of action distribution regularization | Proxy-reward policy optimization · Pandemic mitigation | true reward: -12.29 $\pm$ 0.05 -> -11.17 $\pm$ 0.17 | Helped |
| Use state-action occupancy measure regularization instead of action distribution regularization | Proxy-reward policy optimization · Glucose monitoring | true reward: -74.8 $\pm$ 11.8 -> -\textbf{47.6} $\pm$ 0.6 | Helped |
| Use state-action occupancy measure regularization instead of action distribution regularization | Proxy-reward policy optimization · AI safety gridworld | true reward: 6.24 $\pm$ 0.09 -> \textbf{9.17} $\pm$ 0.11 | Helped |
| Use χ² rather than KL regularization for action distributions | Proxy-reward policy optimization · RLHF | true reward: 16.81 $\pm$ 0.27 -> \textbf{16.94} $\pm$ 0.07 | Helped |
| Use χ² rather than KL occupancy regularization | Proxy-reward policy optimization · AI safety gridworld | true reward: 6.86 $\pm$ 0.17 -> 9.17 $\pm$ 0.11 | Helped |
| Use χ² rather than KL action-distribution regularization | Proxy-reward policy optimization · Traffic control | true reward: -1.33 $\pm$ 0.05 -> -1.29 $\pm$ 0.10 | No clear effect |
| Use χ² rather than KL action-distribution regularization | Proxy-reward policy optimization · Pandemic mitigation | true reward: -12.20 $\pm$ 0.06 -> -12.29 $\pm$ 0.05 | No clear effect |
| Use χ² rather than KL action-distribution regularization | Proxy-reward policy optimization · Glucose monitoring | true reward: -73.4 $\pm$ 8.3 -> -74.8 $\pm$ 11.8 | No clear effect |
| Use χ² rather than KL action-distribution regularization | Proxy-reward policy optimization · AI safety gridworld | true reward: 6.33 $\pm$ 0.11 -> 6.24 $\pm$ 0.09 | No clear effect |
| Optimize the proxy reward without regularization | Proxy-reward policy optimization · Traffic control | true reward: -2.28 $\pm$ 0.00 -> -57.38 $\pm$ 3.53 | Hurt |
| Optimize the proxy reward without regularization | Proxy-reward policy optimization · Pandemic mitigation | true reward: -12.26 $\pm$ 0.00 -> -29.57 $\pm$ 6.86 | Hurt |
Surprising findings
- Action-distribution regularization can provably fail to lower-bound true reward improvement for any f-divergence form
- Lower bound on true-reward gain decreases as proxy correlation weakens, counter to intuition
Limitations
- In RLHF, AD and OM regularization are equivalent only under the contextual bandit assumption (gamma=0)
- Theorem gives a lower bound that may not be positive in general; improvement is empirical
- Experiments use simulated environments and a small RLHF setup rather than large-scale LLM training
Experimental setup
Datasets: TrafficSimulator PandemicSimulator SimGlucose AI safety tomato-watering gridworld RLHF preference reward models
Benchmarks: traffic control pandemic mitigation glucose monitoring RLHF AI safety gridworld
Metrics: true reward (median over 5 seeds)
Baselines: KL action distribution regularization no regularization training with true reward reference policy
Builds on: PPO KL-regularized RLHF Goodhart's law formalisms DPO
Closest papers by meaning
- The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking (ICML 2025 · similarity 0.90)
- Reward Hacking in Rubric-Based Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.89)
- InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling (NeurIPS 2024 · similarity 0.89)
- Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (ICML 2026 · similarity 0.89)
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.