Research map / Reinforcement learning / Visual and representation RL
Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cluster Edge Exploration (CE²), a model-based Go-Explore method that clusters reachable states in a temporal-distance latent space and sends the agent to cluster-boundary goals chosen by world-model exploration potential.
Problem
Efficient exploration in unsupervised goal-conditioned RL: frontier-based Go-Explore goal selection (MEGA, PEG) often picks rare frontier goals the current policy cannot reach, reducing exploration.
Approach
Learn a latent space shaped by a temporal-distance loss, fit a GMM over replay-buffer latent states so clusters capture mutually reachable states, sample low-probability points at cluster edges as goals, and rank them by PEG-style exploration potential computed in a world model.
Key results
- On 3-block stacking, CE²-G achieves a success rate above 90%, while MEGA-G, PEG-G and GC-Dreamer stay below 40%.
- CE² outperforms PEG and MEGA in learning performance and speed in all tasks except PointMaze, where it performs comparably.
- Ablation: CE²-noPEG (edge sampling only) performs well across tasks and beats MEGA and MEGA-wPEG; CE² is insensitive to the number of clusters N_c.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use cluster-edge goal selection with exploration-potential filtering | CE² · Ant Maze and 3-Block Stacking | goal-reaching success rate | Helped |
| use cluster-edge goal selection with exploration-potential filtering | CE² · Point Maze | goal-reaching success rate | No clear effect |
| use CE²-G instead of GC-Dreamer, PEG-G, or MEGA-G | goal-conditioned exploration methods · 3-Block Stacking | success rate: exceeding 90%; less than 40% | Helped |
| add exploration-potential filtering to cluster-edge goal selection (ablation) | CE²-noPEG · Block Rotation | goal-reaching success rate | Hurt |
| remove exploration-potential filtering and randomly select a cluster-edge goal (ablation) | CE² · 3-Block Stacking and Ant Maze | goal-reaching success rate | Hurt |
| remove exploration-potential filtering and randomly select a cluster-edge goal (ablation) | CE² · Block Rotation | goal-reaching success rate | Helped |
| replace cluster-edge candidate goals with unrestricted goals selected by exploration potential (ablation) | CE² · 3-Block Stacking and Ant Maze | goal-reaching success rate | Hurt |
| use latent cluster-edge goals instead of low-density replay-buffer goals (ablation) | MEGA · reported ablation environments | goal-reaching success rate | Helped |
| add PEG filtering to MEGA's low-density candidate goals (ablation) | MEGA · reported ablation environments | goal-reaching success rate | No clear effect |
| vary the number of latent state clusters (ablation) | CE² · reported ablation environments | goal-reaching success rate | No clear effect |
Surprising findings
- Block Rotation is the only environment where CE²-noPEG outperforms full CE², because CE² agents often pursue states where the block falls from the palm.
- GC-Dreamer, with no Go-Explore phase, sometimes outperforms MEGA-G and PEG-G (block and pen rotation).
Limitations
- Requires a well-trained latent space and an accurate temporal distance predictor, adding computational cost.
- Relies on world models (Dreamer) which are more computationally demanding than model-free methods.
- Evaluated only in simulation on six environments; not tested on harder tasks like peg insertion or fluid tasks.
Experimental setup
Datasets: simulated robotics environments: Point Maze, Ant Maze, Walker, 3-Block Stacking, Block Rotation, Pen Rotation
Benchmarks: Point Maze Ant Maze Walker 3-Block Stacking Block Rotation Pen Rotation
Metrics: goal-reaching success rate
Baselines: PEG MEGA GC-Dreamer PEG-G MEGA-G L3P MEGA-wPEG CE²-noPEG
Builds on: Dreamer LEXA Plan2Explore PEG Go-Explore L3P
Closest papers by meaning
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.88)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.88)
- Exploration by Learning Diverse Skills through Successor State Representations (NeurIPS 2024 · similarity 0.88)
- Learning World Models for Unconstrained Goal Navigation (NeurIPS 2024 · similarity 0.87)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.