Research map / Reinforcement learning / Visual and representation RL
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase policy model capacity | RL agents trained on misspecified proxy rewards · Three selected environments | true reward | Hurt |
| Increase training steps | Atari Riverraid agent · Riverraid with underpenalized smoothness | true reward | Hurt |
| Increase action space resolution | Traffic RL agent · Traffic environment | true reward | Hurt |
| Increase random testing rate and observation fidelity | COVID RL agent · COVID ontological misspecification | true reward | Hurt |
| Increase policy model capacity | Riverraid agent optimizing the pacifist proxy · Riverraid with shooting penalty | true reward | Helped |
| Use mean action-distribution distance instead of range Hellinger | Polynomaly anomaly detectors · Traffic-Mer ontological | AUROC: 71.4% vs. 52.7% | Hurt |
| Use range Hellinger instead of mean Jensen-Shannon | Polynomaly anomaly detectors · COVID ontological | AUROC: 45.2% -> 88.1% | Helped |
| Use mean Jensen-Shannon instead of range Hellinger | Polynomaly anomaly detectors · Traffic-Mer scope | AUROC: 57.1% -> 74.6% | Helped |
| Use mean Jensen-Shannon instead of range Hellinger | Polynomaly anomaly detectors · Traffic-Bot misweighting | AUROC: 74.1% -> 88.9% | Helped |
Closest papers by meaning
- Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (ICML 2026 · similarity 0.88)
- Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.88)
- InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling (NeurIPS 2024 · similarity 0.88)
- Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatch (NeurIPS 2024 · similarity 0.86)
- The Value of Reward Lookahead in Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Novelty Detection in Reinforcement Learning with World Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.