Research map / Reinforcement learning / RL from feedback
Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that under heavy-tailed reward error, KL-regularized RLHF policies can reach arbitrarily high proxy reward with vanishing KL while true utility stays at base level ("catastrophic Goodhart"), and tests reward model tails empirically finding them consistent with light tails.
Problem
Whether KL regularization from a base model can keep RLHF outcomes good when the learned reward has misspecification error, particularly when that error is heavy-tailed.
Approach
Shows theoretically that heavy-tailed reward error lets policies with vanishing KL divergence from the base model obtain arbitrarily high proxy reward with no true utility gain, while light-tailed independent error yields high utility under KL penalties; then measures tails of open reward models (Pythia-1.4B, Starling-7B) and finds them consistent with light tails.
Key results
- Theorems show that for any heavy-tailed reference reward distribution and any M, epsilon there is a policy with mean reward above M and KL below epsilon.
- Starling-7B-alpha reward on random and Llama-generated text looks approximately normal, with Hill estimates consistent with light tails.
- Adversarial ACG optimization of Starling gave a KL of 13.35 nats for only about 0.026 reward gain, far below what conditioning achieves at the same KL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply a KL penalty when proxy reward error is heavy-tailed | RLHF policy optimization · Heavy-tailed reward error | true utility | Hurt |
| Use a KL penalty when reward error is light-tailed and independent of utility | RLHF policy optimization · Light-tailed independent error | true utility | Helped |
| Condition on increasingly high proxy reward with heavy-tailed error | Proxy-reward optimization · Heavy-tailed error | true utility | No clear effect |
| Condition on increasingly high proxy reward with light-tailed independent error | Proxy-reward optimization · Light-tailed independent error | true utility | Helped |
| Use ACG to search for high-reward token sequences | Starling 7B-alpha · ACG-generated inputs | reward: 2.2377-0.3329; 0.02571 | Mixed |
| Sample random token sequences and inspect reward tails | Pythia 1.4B reward model · Random length-1024 sequences | reward-tail evidence | No clear effect |
| Sample Llama-generated sequences and inspect reward tails | Starling 7B-alpha · Llama 7B-chat-generated sequences | reward-tail evidence | No clear effect |
| Use a 1% frequency of the highest-reward ACG input instead of the base model | Llama 2-chat · Base reward comparison | reward increase over base model: 0.02571 | Helped |
Surprising findings
- Intuitively reasonable independence-free weakenings of the light-tailed assumption can make the utility limit zero, so KL control can fail even with apparently mild assumptions.
- Open reward models appear light-tailed, which contrasts with prior overoptimization findings for small reward models.
Limitations
- Only a limited search over token sequences was done, so heavy tails cannot be ruled out for the reward models.
- Theorems for light-tailed error assume independence between error and utility, which likely fails in practice.
- Tail measurements use only two reward models and sampling-based evidence.
Experimental setup
Datasets: random token sequences Llama 7B-chat generated sequences
Benchmarks: RewardBench
Metrics: Hill estimator Q-Q plots (normal and exponential) KL divergence reward distribution tail shape
Baselines: Greedy Coordinate Gradient
Builds on: RLHF (Ziegler et al.) Gao et al. 2023 scaling reward model overoptimization Laidlaw et al. state occupancy regularization Greedy Coordinate Gradient
Closest papers by meaning
- Logarithmic Regret for Online KL-Regularized Reinforcement Learning (ICML 2025 · similarity 0.91)
- The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret (ICML 2025 · similarity 0.89)
- Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective (ICML 2025 · similarity 0.89)
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.89)
- Goodhart's Law in Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.