Research map / Reinforcement learning / Visual and representation RL
Goal Misgeneralization in Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with randomly placed coins in 2% of levels (ablation) | PPO agent on CoinRun · CoinRun with randomized coin positions at test time | correct goal generalization | Helped |
| Increase the fraction of training levels with randomly placed coins (ablation) | PPO agent on CoinRun · CoinRun with randomized coin positions at test time | goal misgeneralization frequency | Helped |
| Increase the size of the region in which training goals can appear (ablation) | PPO agent on Maze · Maze with fully randomized goals at validation | validation performance | Helped |
| Train to reach a fixed goal location rather than varying goal locations | PPO agent on Maze · Maze with randomly located cheese at test time | correct goal generalization | Hurt |
| Train on a yellow line, leaving color and shape cues ambiguous at test time | PPO agent on Maze II · Yellow gem versus red line | frequency choosing yellow gem: 89% | Mixed |
| Train with many chests and relatively few keys, then test with many keys and few chests | PPO agent on Keys and Chests · Keys and Chests, many-keys test distribution | time-discounted return | Hurt |
| Use a permeable end wall at test time | PPO actor on CoinRun · CoinRun, permeable end wall | frequency passing through wall: 100% | Helped |
| Evaluate the critic with the coin present versus absent | PPO critic on CoinRun · CoinRun | critic value sensitivity to coin presence | No clear effect |
| Use actor-and-device mixture likelihoods to classify trajectories | Agent/device formalism · Gridworld | distinguishing goal misgeneralization from capability failure | Helped |
Closest papers by meaning
- Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning (AAAI 2022 · similarity 0.85)
- Goal-Conditioned Generators of Deep Policies (AAAI 2023 · similarity 0.84)
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (NeurIPS 2016 · similarity 0.84)
- Hierarchical Goal Abstractions via Learned Subset Relations (ICML 2026 · similarity 0.84)
- Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization for Goal-Conditioned Reinforcement Learning (ICML 2025 · similarity 0.84)
- Horizon Generalization in Reinforcement Learning (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.