Research map / Reinforcement learning / Visual and representation RL
Improving Intrinsic Exploration by Creating Stationary Objectives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SOFE, which augments the agent's state with sufficient statistics of count-based, pseudo-count (E3B), and state-entropy exploration bonuses so the intrinsic rewards become stationary functions of the state.
Problem
Exploration bonuses such as count-based rewards, pseudo-counts, and state-entropy maximization are non-stationary because their sufficient statistics change during training, making the induced POMDP hard to optimize with Markovian policies.
Approach
Augment observations with the sufficient statistics of the intrinsic reward (state-visitation counts, the E3B ellipsoid matrix, or the Gaussian mean/variance for S-Max), turning the bonus into a stationary deterministic function of the augmented state and allowing standard RL agents to optimize it end-to-end.
Key results
- On DeepSea 30, SOFE-DQN reaches 0.42 ± 0.33 versus 0.10 ± 0.10 for DeRL-DQN, and SOFE-DQN is best on DeepSea 20, 24 and 30.
- SOFE enables PPO, PPO+LSTM and A2C to reach near-optimal goal-reaching performance in Maze 2 where vanilla count-based bonuses do not.
- SOFE improves E3B on Procgen-Maze, where E3B only consistently reaches goals when using SOFE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| augment DQN with SOFE state statistics instead of using DeRL | DQN · DeepSea 20, DeepSea 24, DeepSea 30 | average performance: 0.40 ± 0.08 -> 0.70 ± 0.28; 0.53 ± 0.27 -> 0.65 ± 0.26; 0.10 ± 0.10 -> 0.42 ± 0.33 | Helped |
| augment DQN with SOFE state statistics instead of using DeRL | DQN · DeepSea 10, DeepSea 14 | average performance: 0.98 ± 0.09 -> 0.97 ± 0.29; 0.95 ± 0.17 -> 0.78 ± 0.21 | Hurt |
| augment A2C with SOFE state statistics instead of using DeRL | A2C · DeepSea 10, DeepSea 14, DeepSea 20, DeepSea 24, DeepSea 30 | average performance: 0.98 ± 0.10 -> 0.94 ± 0.19; 0.65 ± 0.23 -> 0.45 ± 0.31; 0.42 ± 0.16 -> 0.11 ± 0.25; 0.07 ± 0.10 -> 0.08 ± 0.14; 0.09 ± 0 | Hurt |
| augment PPO with SOFE state statistics instead of using DeRL | PPO · DeepSea 10, DeepSea 14, DeepSea 20, DeepSea 24, DeepSea 30 | average performance: 0.61 ± 0.20 -> 0.77 ± 0.29; 0.92 ± 0.18 -> 0.67 ± 0.33; -0.01 ± 0.01 -> 0.13 ± 0.09; 0.63 ± 0.27 -> 0.07 ± 0.15; -0.01 ± | Mixed |
| augment count-based rewards with SOFE state statistics | count-based exploration · reward-free mazes | map coverage | Helped |
| augment count-based rewards with SOFE state statistics | SAC · Godot 3D map | map coverage | Helped |
| augment the state-entropy objective with SOFE statistics | S-Max · Maze 2 | episodic map coverage | Helped |
| augment count-based rewards with SOFE state statistics | PPO, PPO-LSTM, and A2C · Maze 2 sparse-reward goal task | extrinsic reward | Helped |
| augment E3B with SOFE state statistics | E3B · MiniHack-MultiRoom-N6-v0 | extrinsic reward | Helped |
| augment E3B with SOFE state statistics | E3B · Procgen-Maze | extrinsic reward | Helped |
| augment E3B with SOFE state statistics | E3B · Habitat HM3D held-out scenes | map coverage | Helped |
Surprising findings
- On DeepSea, the simpler SOFE-A2C and SOFE-PPO are not uniformly better than DeRL; DeRL-A2C and DeRL-DQN match or beat SOFE on DeepSea 10 and 14.
Limitations
- Sufficient statistics are only exact in finite or grid environments; for pixel inputs, approximations (e.g. E3B ellipsoid as image) are needed.
- Gains are not uniform across all settings; DeRL matches or beats SOFE in some DeepSea variants.
Experimental setup
Datasets: HM3D MiniHack MultiRoom-N6 Procgen-Maze DeepSea Godot 3D map
Benchmarks: MiniHack MultiRoom-N6 Procgen-Maze DeepSea HM3D Godot 3D map 3 custom 32x32 mazes
Metrics: map coverage interquantile mean of extrinsic reward average return
Baselines: vanilla count-based bonus S-Max DeRL E3B ICM RND
Builds on: E3B DeRL count-based exploration IMPALA
Closest papers by meaning
- Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus (NeurIPS 2024 · similarity 0.88)
- CIG: Exploration via Conditional Information Gain (NeurIPS 2026 · similarity 0.87)
- Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning (ICML 2024 · similarity 0.87)
- MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization (ICLR 2025 · similarity 0.87)
- Random Latent Exploration for Deep Reinforcement Learning (ICML 2024 · similarity 0.86)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.