Research map / Reinforcement learning / Visual and representation RL
Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Effective Metric-based Exploration-bonus (EME), a bisimulation-free state-discrepancy metric with KL policy term and an ensemble-variance scaling factor, as an intrinsic exploration bonus for hard-exploration RL tasks.
Problem
Metric-based state-discrepancy exploration bonuses rely on episodic count scaling factors that fail when states are unique, and bisimulation-based metrics (e.g., LIBERTY) suffer from approximation gaps that break theoretical guarantees and degrade in hard exploration tasks.
Approach
Replace the Wasserstein-based bisimulation metric with an EME metric that strictly computes reward-difference expectations (via ensemble-based variance correction), adds a KL divergence between policy distributions, and scales the bonus by the variance of an ensemble of reward model predictions instead of episodic counts.
Key results
- On MiniGrid, EME achieves 0.67 on MRN7S8 and 0.95 on OM2D1h, solving all exploration tasks and best in 4 of 6 settings.
- On Robosuite, EME consistently outperforms ICM, RND, RIDE, NovelD, E3B and LIBERTY across Door Opening, Table Wiping and Pick-and-Place.
- On delayed-reward MuJoCo, EME achieves best or second-best in 15 of 18 tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the episodic count term from RIDE (ablation) | RIDE · Robosuite | success rate: 0.52 -> 0.05 | Hurt |
| remove the episodic count term from RIDE (ablation) | RIDE · MiniGrid | success rate: 0.93 -> 0.00 | Hurt |
| remove the episodic count term from RIDE (ablation) | RIDE · Habitat | success rate: 0.19 -> 0.00 | Hurt |
| remove the episodic count term from NovelD (ablation) | NovelD · Robosuite | success rate: 0.55 -> 0.11 | Hurt |
| remove the episodic count term from NovelD (ablation) | NovelD · MiniGrid | success rate: 0.93 -> 0.00 | Hurt |
| remove the episodic count term from NovelD (ablation) | NovelD · Habitat | success rate: 0.17 -> 0.00 | Hurt |
| use EME instead of RIDE | EME and RIDE · MRN7S8 | success rate: 0.67 ± 0.001 -> 0.67 ± 0.011 | No clear effect |
| use EME instead of RIDE | EME and RIDE · MRN7S8-NT | success rate: 0.65 ± 0.001 -> 0.65 ± 0.003 | No clear effect |
| use EME instead of RIDE | EME and RIDE · KCS4R3 | success rate: 0.93 ± 0.002 -> 0.91 ± 0.011 | Hurt |
| use EME instead of RIDE | EME and RIDE · KCS4R3-NT | success rate: 0.81 ± 0.001 -> 0.91 ± 0.003 | Helped |
| use EME instead of NovelD | EME and NovelD · OM2D1h | success rate: 0.96 ± 0.011 -> 0.95 ± 0.028 | Hurt |
| use EME instead of NovelD | EME and NovelD · OM2D1h-NT | success rate: 0.91 ± 0.008 -> 0.95 ± 0.011 | Helped |
Surprising findings
- LIBERTY scores 0.00 on several MiniGrid settings despite strong performance elsewhere, and RIDE/NovelD drop to 0.00 without the episodic count term.
- Ensemble-variance-based scaling with a larger ensemble (12 models) degraded performance, attributed to smaller variance reducing bonus scale.
Limitations
- Performance drops with bisimulation or inverse-dynamics encoders because bonuses shrink quickly.
- Computational cost grows with reward-model ensemble size.
Experimental setup
Datasets: Atari hard exploration games MiniGrid Robosuite Habitat (Matterport3D) MuJoCo
Benchmarks: Robosuite Atari MiniGrid Habitat MuJoCo delayed reward
Metrics: success rate average return exploration coverage mean success rate
Baselines: ICM RND RIDE NovelD E3B LIBERTY
Builds on: LIBERTY RIDE NovelD RND bisimulation metric
Closest papers by meaning
- Efficient Potential-based Exploration in Reinforcement Learning using Inverse Dynamic Bisimulation Metric (NeurIPS 2023 · similarity 0.89)
- Episodic Novelty Through Temporal Distance (ICLR 2025 · similarity 0.89)
- Task-Aware Exploration via a Predictive Bisimulation Metric (ICML 2026 · similarity 0.89)
- Improving Intrinsic Exploration by Creating Stationary Objectives (ICLR 2024 · similarity 0.88)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- CIG: Exploration via Conditional Information Gain (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.