Research map / Reinforcement learning / Policy gradient and actor-critic
The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically shows that low expected test error of a learned reward model does not guarantee low policy regret, characterizing safe vs. unsafe data distributions for unregularized, KL-regularized, and RLHF policy optimization.
Problem
Reward learning yields reward models with low error on the training data distribution, yet policies optimized on them can have large true regret due to distributional shift during policy optimization. The paper asks when low reward-model error guarantees low regret.
Approach
Formalizes "error-regret mismatch" and "safe/unsafe data distributions" in a worst-case framework: proves low error suffices for positive-coverage distributions only with prohibitively small epsilon, gives necessary and sufficient linear constraints for unregularized optimization, and shows regularization (including KL/RLHF) does not remove the failure mode.
Key results
- Low expected reward error epsilon guarantees low worst-case regret for positive data distributions, but the required epsilon scales with the minimum of D(s,a), so it can be prohibitively small.
- For any fixed epsilon, data distributions giving the support of a bad policy total probability at most epsilon/(1+C) are unsafe (Theorem on regularized optimization).
- Corollary: if there are at least 1/epsilon policies with regret >= L and disjoint supports, every data distribution is unsafe.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a data distribution with low coverage of a high-regret policy while keeping reward-model expected error at ε | Unregularized policy optimization · General MDPs | regret: D(supp D̂π) < ε; RegR(π̂) ≥ L | Hurt |
| Reduce coverage of a deterministic worst-case policy under the training distribution | Regularized policy optimization · General MDPs | regret: D(supp Dπ*) ≤ ε/(1 + C); RegR(π̂) ≥ L | Hurt |
| Give a bad action sufficiently low probability under the reference policy | KL-regularized RLHF · Contextual bandits | regret: RegR(π̂) ≥ L | Hurt |
| Increase the number of high-regret policies with disjoint support until |ΠL| ≥ 1/ε | Unregularized policy optimization · General MDPs | safety of data distributions: |ΠL| ≥ 1/ε; unsafeD = DistributionsOverSet(S × A) | Hurt |
| Reduce reward-model error sufficiently for a positive data distribution | Unregularized policy optimization · General MDPs | regret: ε < (1 − γ)/√2 · (range JR/range R) · min(s,a) D(s,a) · L | Helped |
| Reduce expected reward-model error toward zero | Reward learning with policy optimization · General MDPs | worst-case regret | Helped |
Surprising findings
- When many bad policies have disjoint support, every data distribution is unsafe, even with arbitrarily small expected reward error.
- KL-regularization toward a reference policy does not prevent error-regret mismatch; RLHF with a reference policy giving low probability to bad actions can still yield regret >= L.
Limitations
- Worst-case framework over reward models and optimal policies; does not model inductive biases of optimization procedures or non-optimal policies.
- Negative results rely on constructed adversarial reward models; computing the characterizing matrix M is infeasible in practice.
Experimental setup
Metrics: regret
Builds on: RLHF Berge's maximum theorem offline RL coverage conditions
Closest papers by meaning
- STARC: A General Framework For Quantifying Differences Between Reward Functions (ICLR 2024 · similarity 0.89)
- Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification (NeurIPS 2024 · similarity 0.89)
- The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards (AAAI 2025 · similarity 0.89)
- Learning Optimal Advantage from Preferences and Mistaking It for Reward (AAAI 2024 · similarity 0.88)
- A Theoretical Framework for Partially-Observed Reward States in RLHF (ICLR 2025 · similarity 0.88)
- Logarithmic Regret for Online KL-Regularized Reinforcement Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.