Research map / Reinforcement learning / Visual and representation RL
Catching Two Birds with One Stone: Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DuRND, a reward-shaping framework using two random network distillation modules (positive and negative) to jointly produce a novelty reward and a contribution reward for PPO in sparse-reward RL.
Problem
Sparse-reward reinforcement learning where novelty bonuses cause distraction from task objectives while hidden-state value shaping lacks early exploration; balancing exploration and exploitation.
Approach
Two lightweight RND modules are trained on states preceding environmental-reward anchor states (positive) versus all other states (negative); their prediction errors combine into a novelty reward (sum of errors) and a contribution reward (negative-module error normalized by total error, i.e., estimated probability of being positive), shaping PPO's rewards.
Key results
- DuRND-adp reaches 6654.04 on MontezumaRevenge vs 5459.01 for NGU and 1337.34 for RND (100-episode test returns).
- DuRND-adp achieves 113.15 on Pitfall and 42.63 on Solaris, outperforming the best baseline on most of the twelve tasks in Table 1.
- Memory consumption of baselines such as ExploRS and ROSA is roughly 10-11x that of DuRND, while RND is about 0.9x and PPO about 0.8x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DuRND-adp instead of the strongest baseline | PPO · MontezumaRevenge | average episodic return: 5459.01±0.08 -> 6654.04±5.26 | Helped |
| Use DuRND-adp instead of the strongest baseline | PPO · Pitfall | average episodic return: 95.40±0.03 -> 113.15±0.03 | Helped |
| Use DuRND-adp instead of the strongest baseline | PPO · Frogger | average episodic return: 14.80±0.00 -> 14.82±0.00 | Helped |
| Use DuRND-adp instead of the strongest baseline | PPO · Freeway | average episodic return: 24.94±0.00 -> 25.53±0.00 | Helped |
| Use DuRND-adp instead of the strongest baseline | PPO · Solaris | average episodic return: 33.74±0.00 -> 42.63±0.00 | Helped |
| Use DuRND-adp instead of the strongest baseline | PPO · BeamRider | average episodic return: 29.21±0.02 -> 34.13±0.01 | Helped |
| Use DuRND-adp instead of the strongest baseline | PPO · DefendLine | average episodic return: 10.22±0.00 -> 11.03±0.00 | Helped |
| Use DuRND-adp instead of the strongest baseline | PPO · SaveCenter | average episodic return: 11.87±0.00 -> 14.01±0.00 | Helped |
| Use DuRND-adp instead of the strongest baseline | PPO · CollectKit | average episodic return: 35.53±0.00 -> 37.34±0.01 | Helped |
| Use DuRND-adp instead of the strongest baseline | PPO · SlayGhosts | average episodic return: 17.29±0.01 -> 18.29±0.00 | Helped |
| Use DuRND-adp instead of the strongest baseline | PPO · ThreeRooms | average episodic return: 0.99±0.00 -> 0.97±0.00 | No clear effect |
| Use DuRND-adp instead of the strongest baseline | PPO · TMaze | average episodic return: 1.00±0.00 -> 1.00±0.00 | No clear effect |
Surprising findings
- Vanilla RND is markedly weaker than DuRND on most tasks, and using only the novelty term of DuRND still outperforms vanilla RND.
Limitations
- Relies on environmental sparse rewards to label positive and negative states, so limited for dense-reward tasks.
- Positive sequence length T_pos is set by fixed or linear-schedule heuristics rather than adapted automatically to reward sparsity.
Experimental setup
Datasets: MontezumaRevenge Pitfall Frogger Freeway Solaris BeamRider DefendLine SaveCenter CollectKit SlayGhosts ThreeRooms TMaze
Benchmarks: MontezumaRevenge Pitfall Frogger Freeway Solaris BeamRider DefendLine SaveCenter CollectKit SlayGhosts ThreeRooms TMaze
Metrics: average episodic return standard error memory consumption relative to DuRND
Baselines: ExploRS NGU DEIR RND #Explo ReLara ROSA SORS PPO
Builds on: Random Network Distillation PPO RLeXplore ReLara
Closest papers by meaning
- Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Reward Shaping for Reinforcement Learning with An Assistant Reward Agent (ICML 2024 · similarity 0.88)
- Action-Dependent Optimality-Preserving Reward Shaping (ICML 2025 · similarity 0.88)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Exploration and Anti-Exploration with Distributional Random Network Distillation (ICML 2024 · similarity 0.87)
- ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.