Research map / Reinforcement learning / Visual and representation RL
Bootstrapped Reward Shaping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BSRS, a bootstrapped potential-based reward shaping method where the agent's current state-value estimate serves as the potential, with convergence proofs in tabular settings and gains on Atari (DQN) and Pendulum (TD3).
Problem
Sparse-reward RL needs many environment steps; PBRS speeds learning but requires a hand-designed task-specific potential, while the optimal-value potential is circular since it requires knowing the solution.
Approach
Use the agent's current estimate of the optimal state-value function, recomputed each training step, as the PBRS potential (Phi = eta * max_a Q), so shaping is adaptive and needs only a one-line change to value-based algorithms; a scalar shape-scale eta replaces the function-class search.
Key results
- On Atari (40 games, 10M steps), BSRS with eta around 2 raises aggregate median human-normalized score from about 45% to 60% over DQN.
- BSRS improved DQN by more than 10% in 21 of 40 Atari environments and by over 100% in 5 of 40; it hurt performance in 6 of 40.
- In the tabular gridworld, shaping with eta roughly 3 times beyond the proven bound reduced steps to convergence by about 20%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BSRS using the current value estimate as the PBRS potential | DQN · Atari suite, 40 environments | median human-normalized score: 45%→60% | Helped |
| Add BSRS using the current value estimate as the PBRS potential | DQN · Atari suite, 40 environments | performance: 21/40 environments; more than 10% | Helped |
| Add BSRS using the current value estimate as the PBRS potential | DQN · Atari suite, 40 environments | performance: 5/40 environments; over 100% | Helped |
| Add BSRS using the current value estimate as the PBRS potential | DQN · Atari suite, 40 environments | performance: 6/40 environments | Mixed |
| Tune η to an intermediate value around 2 (ablation) | BSRS with DQN · Atari suite | median human-normalized score: η≈2 | Mixed |
| Use negative η (ablation) | BSRS with DQN · Atari suite | performance: η=10 | Hurt |
| Increase η beyond the tested intermediate optimum (ablation) | BSRS with DQN · Atari suite | performance: η≈2 | Hurt |
| Use the online network rather than the target network to calculate Φ (ablation) | BSRS with DQN · Atari suite | performance | Helped |
| Use the target network rather than the online network to calculate Φ (ablation) | BSRS with DQN · Atari suite | performance | Hurt |
| Add BSRS to TD3, using the actor-sampled action to calculate the potential | TD3 · Pendulum-v1 | reward | Helped |
| Use negative η (ablation) | BSRS with TD3 · Pendulum-v1 | performance | Hurt |
| Add BSRS with an optimized η | TD3 · Hopper-v4 | performance | No clear effect |
Surprising findings
- Using the online network rather than the target network to compute the potential substantially improved performance, contrary to the stability intuition.
- Shape-scale values beyond the proven contraction range (eta in (-1, (1-gamma)/(1+gamma))) still improved tabular convergence, and negative eta performed worse than large positive eta in deep RL.
Limitations
- Convergence is proven only for the tabular setting; deep RL results rely on empirical evidence and the bound on eta is not tight in practice.
- Continuous-action potential uses the actor's sampled action rather than a true maximum over actions, an approximation.
- Gains on more complex continuous tasks (Humanoid-v4) were not statistically significant; compute cost is large (about 1600 GPU-days for Atari sweep).
Experimental setup
Datasets: Arcade Learning Environment (40 Atari games) Pendulum-v1 7x7 sparse-reward gridworld Hopper-v4 Humanoid-v4
Benchmarks: Arcade Learning Environment (40 games) Pendulum-v1 7x7 gridworld Hopper-v4 Humanoid-v4
Metrics: median human-normalized score steps to convergence episode reward
Baselines: DQN (eta=0) TD3 (eta=0)
Builds on: potential-based reward shaping DQN TD3 Dynamic PBRS
Closest papers by meaning
- Automatic Reward Shaping from Confounded Offline Data (ICML 2025 · similarity 0.89)
- Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Potential-Based Reward Shaping for Intrinsic Motivation (Student Abstract) (AAAI 2024 · similarity 0.88)
- Reward Shaping for Reinforcement Learning with An Assistant Reward Agent (ICML 2024 · similarity 0.88)
- BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping (ICLR 2025 · similarity 0.87)
- Action-Dependent Optimality-Preserving Reward Shaping (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.