Research map / Reinforcement learning / Imitation learning
The Perils of Trial-and-Error Reward Design: Misdesign through Overfitting and Invalid Task Specifications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| change discount factor from 0.99 to 0.8 | Q-learning with reward [-1.0,-1.0,-1.0,0.1] · Hungry Thirsty 6x6 | cumulative true reward: [4,965; 20,234] -> [86,653; 101,922] | Helped |
| change algorithm from A2C to DDQN | reward [-0.1,0.2,0.5,1.0] · Hungry Thirsty 6x6 | mean cumulative true performance: 107 -> 232055 | Helped |
| vary discount factor across 0.99, 0.8, 0.5 | ranking of 5196 reward functions · Hungry Thirsty | Kendall tau_b: 0.07, 0.04, 0.12 | Hurt |
| vary learning rate from 0.25 to 0.05 | ranking of 5196 reward functions · Hungry Thirsty | Kendall tau_b: 0.11 | Hurt |
| vary algorithm (PPO, A2C, DDQN, Q-learning) | ranking of 107 reward functions · Hungry Thirsty 6x6 | Kendall tau_b: 0.25 for PPO vs A2C; -0.04 to 0.13 otherwise | Mixed |
| use best reward from one context in another | best reward fn of gamma=0.99 vs best of gamma=0.5 · Hungry Thirsty | reward function optimality: 16 of 18 experiments significant | Hurt |
| change algorithm from PPO to A2C | optimal reward function · Hungry Thirsty | optimal reward function identity | No clear effect |
| move food and water from adjacent to opposite corners | expert-designed rewards · Hungry Thirsty user study | share of valid reward functions: 83% -> 47% | Hurt |
| evaluate final reward vs earlier tried rewards under DDQN | expert P20 reward selection · Hungry Thirsty user study | mean cumulative performance: 138,092 -> 1,031 | Hurt |
| reduce grid from 6x6 to 4x4 | expert user study · Hungry Thirsty | valid reward at opposite corners: 4 of 12 -> 10 of 18 | Helped |
Closest papers by meaning
- The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret (ICML 2025 · similarity 0.90)
- STARC: A General Framework For Quantifying Differences Between Reward Functions (ICLR 2024 · similarity 0.90)
- Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.88)
- R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.88)
- To the Max: Reinventing Reward in Reinforcement Learning (ICML 2024 · similarity 0.88)
- ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.