Research map / Reinforcement learning / Visual and representation RL
Exploration and Anti-Exploration with Distributional Random Network Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distributional Random Network Distillation (DRND), which distills a distribution of N random target networks and derives a pseudo-count-like bonus to fix RND's bonus inconsistency, applied to online PPO exploration and offline SAC-DRND anti-exploration.
Problem
RND's intrinsic bonus is inconsistent: initial bonuses are unevenly distributed across states and final bonuses fail to track visitation frequency, which hampers deep exploration in online RL and anti-exploration penalties in offline RL.
Approach
Instead of distilling one fixed random target, the predictor is trained on a randomly sampled target from N fixed random networks; the moments of this target distribution give a mean-deviation bonus (b1) and a pseudo-count-like statistic (b2, approximating 1/sqrt(n)) without extra networks or tables.
Key results
- SAC-DRND achieves an average D4RL MuJoCo score of 86.0 versus 82.6 for SAC-RND and 81.2 for ReBRAC across the ensemble-free comparison, but on AntMaze it averages 76.0 vs ReBRAC's 76.8.
- On Adroit tasks, SAC-DRND improves the average normalized score from 12.9 (SAC-RND) to 23.5, though ReBRAC reaches 58.6.
- Before training, DRND bonus KL divergence to uniform is 0.0070 vs 0.0377 for RND (lower is better), indicating more uniform initial bonuses.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill a distribution of random target networks instead of a single fixed target network | RND bonus · Before training, 100 initial dataset distributions | D_KL(P||U): 0.0377±0.0248 -> 0.0070±0.0063 | Helped |
| Distill a distribution of random target networks instead of a single fixed target network | RND bonus · After training, 100 initial dataset distributions | D_KL(P||1/sqrt(n)): 0.0946±0.0409 -> 0.0476±0.0389 | Helped |
| Use SAC-DRND instead of SAC-RND | Offline SAC · Gym-MuJoCo D4RL | average normalized score: 82.6 -> 86.0 | Helped |
| Use SAC-DRND instead of SAC-RND | Offline SAC · AntMaze D4RL | average normalized score: 60.9 -> 76.0 | Mixed |
| Use DRND instead of RND | PPO exploration · Montezuma's Revenge, Gravitar, and Venture | episodic return | Helped |
| Use DRND instead of other exploration algorithms | PPO exploration · Adroit Pen | episodic return | No clear effect |
| Use DRND instead of other exploration algorithms | PPO exploration · Adroit Hammer and Relocate | episodic return | Helped |
| Use DRND instead of SAC-RND | Offline SAC · Adroit tasks, average score | normalized score: 12.9 to 23.5 | Helped |
| Use DRND instead of SAC-RND | Offline SAC · Adroit tasks, average without expert dataset | normalized score | Mixed |
Surprising findings
- SAC-DRND underperforms ReBRAC on Adroit tasks (average 23.5 vs 58.6) and on AntMaze average (76.0 vs 76.8), so it is not uniformly superior in offline settings.
Limitations
- Theoretical analysis of the bonus (Lemma on linear models) covers only linear predictors and a simplified setting.
- Offline AntMaze and Adroit results do not consistently beat strong baselines such as ReBRAC and MSG.
- Bonus weighting alpha is fixed and environment-dependent per the authors.
Experimental setup
Datasets: D4RL MuJoCo D4RL AntMaze Atari (Montezuma's Revenge, Gravitar, Venture) Adroit Fetch manipulation
Benchmarks: Montezuma's Revenge Gravitar Venture Adroit Fetch D4RL MuJoCo D4RL AntMaze
Metrics: episodic return normalized score KL divergence expected online performance
Baselines: PPO RND CFN ICM SAC TD3+BC CQL IQL SAC-RND ReBRAC SAC-N EDAC
Builds on: Random Network Distillation Coin Flip Network PPO SAC-RND ReBRAC
Closest papers by meaning
- Exploration by random network distillation (ICLR 2019 · similarity 0.92)
- Random Latent Exploration for Deep Reinforcement Learning (ICML 2024 · similarity 0.89)
- Catching Two Birds with One Stone: Reward Shaping with Dual Random Networks for Balancing Exploration and Exploitation (ICML 2025 · similarity 0.87)
- Improving Intrinsic Exploration by Creating Stationary Objectives (ICLR 2024 · similarity 0.85)
- MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization (ICLR 2025 · similarity 0.85)
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.