Research map / Reinforcement learning / Visual and representation RL
Effective Exploration Based on the Structural Information Principles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SI2E, an exploration framework that defines structural mutual information, learns dynamics-relevant state-action embeddings, and maximizes value-conditional structural entropy over an encoding tree as an intrinsic reward.
Problem
Existing information-theoretic exploration in RL models uncertainty of random variables but ignores the inherent structure of state and action spaces, and structural information principles are limited to single-variable graphs.
Approach
Extends structural information to a two-variable structural mutual information over 2-layer approximate binary encoding trees, uses it to learn state-action embeddings (minimize MI with current state, maximize with next state), and builds an intrinsic reward from value-conditional structural entropy over a policy-value-derived encoding tree.
Key results
- On MetaWorld, SI2E with DrQv2 reaches an average success rate of 97.87 after about 64.79K steps, outperforming baselines.
- On DMControl, SI2E improves Pendulum Swingup average reward from 824.17 (VCSE) to 885.50 and Cartpole Swingup from 707.76 to 795.09 (12.34% gain).
- On MiniGrid, A2C+SI2E raises RedBlueDoors success from 79.82% to 85.80% and cuts required steps from 1161.90K to 461.90K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SI2E exploration instead of VCSE | A2C · RedBlueDoors-6x6 | success rate (%): 79.82 ± 7.26 -> 85.80 ± 1.48 | Helped |
| use SI2E exploration instead of VCSE | A2C · RedBlueDoors-6x6 | required steps (K): 1161.90 ± 241.59 -> 461.90 ± 61.53 | Helped |
| use SI2E exploration instead of VCSE | DrQv2 · Door Open | success rate (%): 80.90 ± 10.19 -> 95.77 ± 1.05 | Helped |
| use SI2E exploration instead of VCSE | DrQv2 · Quadruped Walk | average episode reward: 243.74 ± 29.91 -> 399.51 ± 29.05 | Helped |
| use SI2E exploration instead of VCSE | DrQv2 · Cartpole Swingup | average episode reward: 707.76 ± 50.38 -> 795.09 ± 90.49 | Helped |
| replace the SI2E embedding principle with the Dynamic Bottleneck (ablation) | SI2E · MetaWorld and DMControl tasks | final performance | Hurt |
| replace the SI2E intrinsic reward with VCSE (ablation) | SI2E · MetaWorld and DMControl tasks | final performance | Hurt |
| replace the SI2E embedding principle with the Dynamic Bottleneck (ablation) | SI2E · MetaWorld and DMControl tasks | sample efficiency | Hurt |
| replace the SI2E intrinsic reward with VCSE (ablation) | SI2E · MetaWorld and DMControl tasks | sample efficiency | Hurt |
Surprising findings
- DrQv2+SE can fail badly on some DMControl tasks, e.g. Pendulum Swingup average reward of 10.80, far below the DrQv2 baseline of 424.21 without exploration bonus.
Limitations
- Encoding tree restricted to height 2 for computational tractability.
- Evaluated only on navigation, visual manipulation and control benchmarks with small-scale agents.
Experimental setup
Datasets: MiniGrid MetaWorld DeepMind Control Suite
Benchmarks: MiniGrid MetaWorld DeepMind Control Suite
Metrics: success rate required steps to target reward average episode reward
Baselines: A2C+SE A2C+VCSE DrQv2+SE DrQv2+VCSE DrQv2+MADE SI2E-DB SI2E-VCSE
Builds on: Dynamic Bottleneck value-conditional state entropy structural information principles HCSE RE3
Closest papers by meaning
- Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration (NeurIPS 2023 · similarity 0.88)
- On the Role of Information Structure in Reinforcement Learning for Partially-Observable Sequential Teams and Games (NeurIPS 2024 · similarity 0.88)
- Probing in the Dark: State Entropy Maximization for POMDPs (ICLR 2026 · similarity 0.88)
- Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- CIG: Exploration via Conditional Information Gain (NeurIPS 2026 · similarity 0.87)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.