Research map / Reinforcement learning / MDPs and planning
Goodhart's Law in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Quantifies Goodhart's law in RL across random MDPs and reward pairs, gives a geometric (occupancy-polytope) explanation, and derives an early-stopping method with worst-case reward guarantees.
Problem
Optimising an imperfect proxy reward can reduce true reward beyond some optimisation pressure (Goodhart's law); the paper asks when this occurs in MDPs, why, and how to avoid it provably.
Approach
Measure Goodharting with a normalised drop height using the projected angle between rewards (STARC-style metric) and occupancy-measure geometry; explain drops via steepest ascent along the boundary of the occupancy polytope; use the angle bound to derive an early-stopping rule and a pessimistic objective that maximises worst-case true reward.
Key results
- Goodharting (NDH > 0) occurs in 19.3% of 30400 sampled experiments across grid search over environments, rewards, discount factors and optimisation pressures.
- Early stopping prevents Goodharting in all tested cases (NDH = 0), but loses on average between 10% and 44% of achievable true reward depending on environment.
- Normalised drop height, simple integration, CACW, LR and RWI metrics are all highly correlated, supporting NDH as a single Goodharting score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimise a proxy reward across the sampled environments and reward functions | RL policies in the sampled MDP experiments · All sampled experiments | Goodhart drop prevalence: 19.3\% of all experiments | Mixed |
| Apply the early-stopping criterion during proxy-reward optimisation | Early-stopping method · All evaluated cases | NDH: NDH = 0 | Helped |
| Choose MCE rather than Boltzmann Rationality to operationalise optimisation pressure | Early-stopping method · Evaluated environments | reward loss | No clear effect |
| Use a smaller angle-distance bound θ | Early-stopping method · Evaluated environments | reward loss | Helped |
| Apply pessimistic early stopping rather than continuing proxy-reward optimisation | Early-stopping method · Evaluated environments | reward loss: between 10\% and 44\% | Hurt |
Surprising findings
- The early stopping method, although provably avoiding Goodharting, can lose a substantial amount of true reward (10%-44% on average).
- Goodharting is common but not universal: it occurred in only 19.3% of experiments.
Limitations
- Early stopping requires an upper bound theta on the angle between proxy and true reward, which must be known or estimated.
- Computing occupancy measures and the projection onto the polytope is infeasible exactly in large environments; approximated via MCE/BR.
- Early stopping is pessimistic and can lose substantial true reward compared with naive optimisation.
Experimental setup
Datasets: Gridworld Cliff RandomMDP TreeMDP CyclicMDP
Benchmarks: Gridworld Cliff RandomMDP TreeMDP
Metrics: normalised drop height fraction of experiments with Goodhart drop reward lost by early stopping
Baselines: naive optimisation of proxy reward
Builds on: STARC metric Value Iteration Maximal Causal Entropy Boltzmann rationality
Closest papers by meaning
- Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification (NeurIPS 2024 · similarity 0.88)
- Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.87)
- The Value of Reward Lookahead in Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret (ICML 2025 · similarity 0.86)
- Reinforcement Learning with Lookahead Information (NeurIPS 2024 · similarity 0.86)
- STARC: A General Framework For Quantifying Differences Between Reward Functions (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.