Research map / Reinforcement learning / Visual and representation RL
Random Latent Exploration for Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Random Latent Exploration (RLE), which adds a latent-conditioned random reward (dot product of state features and a sampled latent vector) to PPO so policy and value networks condition on the latent, achieving deep exploration without bonus networks.
Problem
Deep RL exploration in high-dimensional state spaces: noise-based exploration (action noise) produces local perturbations that fail on deep-exploration tasks, while bonus-based methods add extra networks and are complex and unstable; no single family consistently wins across tasks.
Approach
Perturb the task reward with a random state-dependent reward F(s,z)=phi(s)·z, resampling latent z per trajectory, and condition the policy and value function on z (UVFA-style) so the random reward is observable; the feature extractor phi is a slowly updated copy of the value network backbone, so no bonus-prediction ensemble is needed.
Key results
- On Atari (57 games), RLE's IQM human-normalized score exceeds PPO, and its probability of improvement over PPO, NoisyNet and RND has lower confidence bound above 0.5, while NoisyNet and RND have POI over PPO below 0.5.
- On IsaacGym (9 tasks), RLE's probability of improvement over PPO and RND has lower confidence bound above 0.5, and RLE is the only method with a statistically significant POI over PPO.
- In FourRoom reward-free exploration, RLE spreads state visitation across all four rooms while PPO visitation concentrates around the starting room.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RLE exploration to PPO | PPO · Atari (57 games) | IQM human-normalized score | Helped |
| add RLE exploration to PPO | PPO · IsaacGym (9 tasks) | aggregate normalized return | Helped |
| add RLE exploration to PPO | PPO · FourRoom sparse-reward | average score: 0 -> 0.6 | Helped |
| change the latent-vector sampling distribution to Uniform([-0.5, 0.5]^d) or isotropic normal (ablation) | RLE · Atari and IsaacGym | performance | No clear effect |
| vary latent dimension to d=2, d=32, or d=128 instead of d=8 (ablation) | RLE · Atari (selected games) | performance | No clear effect |
| remove latent conditioning from the policy and value networks (ablation) | RLE · Atari Venture | performance | Hurt |
| use a shallower feature network for random reward features (ablation) | RLE · IsaacGym | IQM normalized score | No clear effect |
| replace state-dependent random rewards with white-noise rewards (ablation) | RLE · IsaacGym | performance | Hurt |
| use a completely randomly initialized network for the feature extractor (ablation) | RLE · Atari (subset of games) | performance | Hurt |
Surprising findings
- RND underperforms PPO on average across 57 Atari games in this implementation (POI below 0.5), which the authors attribute partly to using a CNN rather than an LSTM.
- RLE does not improve performance on Montezuma's Revenge, a hard-exploration game.
Limitations
- Evaluated only with on-policy PPO; off-policy extension (DQN, SAC) left to future work.
- Does not improve performance on the hard-exploration game Montezuma's Revenge.
- Gains are on average rather than uniform; evaluation limited to Atari, IsaacGym and a toy FourRoom environment, not real-world robotics.
Experimental setup
Datasets: Atari 57 games IsaacGym 9 tasks FourRoom gridworld
Benchmarks: Atari (57 games) IsaacGym (9 tasks) FourRoom
Metrics: interquartile mean of human-normalized score probability of improvement capped human normalized score PPO-normalized score state visitation counts
Baselines: PPO NoisyNet RND PPO with reward normalization
Builds on: PPO UVFA RND NoisyNet
Closest papers by meaning
- Generalization and Exploration via Randomized Value Functions (ICML 2016 · similarity 0.91)
- Exploring through Random Curiosity with General Value Functions (NeurIPS 2022 · similarity 0.89)
- Exploration and Anti-Exploration with Distributional Random Network Distillation (ICML 2024 · similarity 0.89)
- MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization (ICLR 2025 · similarity 0.88)
- The Effective Horizon Explains Deep RL Performance in Stochastic Environments (ICLR 2024 · similarity 0.88)
- Uncertainty-Aware Reward-Free Exploration with General Function Approximation (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.