Research map / Reinforcement learning / Visual and representation RL
Learning World Models for Unconstrained Goal Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MUN, a goal-directed exploration method for model-based GCRL that trains a Dreamer-based world model on transitions between automatically discovered key subgoals (via distinct action discovery) in both directions, improving world model reliability and generalization to unseen goals.
Problem
Learned world models in goal-conditioned model-based RL generalize poorly to state transitions backward along recorded trajectories or across different trajectories, causing hallucinated rollouts and ineffective exploration under sparse rewards.
Approach
Sample key subgoal states from the replay buffer using Distinct Action Discovery (FPS over action variation), then navigate sequentially between pairs of subgoals during data collection so the world model learns bidirectional transitions between arbitrary buffer states, and train the policy in imagination on this model.
Key results
- MUN achieves over 95% success on 3-Block Stacking within 2.5M steps, while baselines reach only around 60%.
- MUN outperforms GC-Dreamer, PEG-G and MEGA-G across all six environments in final success rate and learning speed (5 seeds).
- On 2250 evaluation trajectories of arbitrary subgoal-to-subgoal navigation in 3-Block Stacking, MUN achieves the highest success rate among methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MUN's bidirectional replay buffer and navigation between replay-buffer subgoals | GC-Dreamer · 3-Block Stacking | success rate: around 60% -> over 95% | Helped |
| use MUN's exploration strategy instead of Go-Explore baselines | GC-Dreamer · all tasks | success rate | Helped |
| set the number of navigation subgoals Ns to 3 instead of 2 (ablation) | MUN · evaluated tasks | training performance | Hurt |
| replace DAD with fixed-time-interval subgoal sampling (ablation) | MUN · 3-Block Stacking | success rate | Hurt |
| replace DAD with fixed-time-interval subgoal sampling (ablation) | MUN · Block Rotation; Pen Rotation | success rate | No clear effect |
| discover subgoals from latent-state cluster centroids instead of distinct actions (ablation) | MUN · 3-Block Stacking | success rate | Hurt |
| discover subgoals from latent-state cluster centroids instead of distinct actions (ablation) | MUN · Block Rotation; Pen Rotation | success rate | No clear effect |
| use DAD to discover key subgoals (ablation) | MUN · 3-Block Stacking | success rate | Helped |
Surprising findings
- Go-Explore baselines do not always improve exploration; GC-Dreamer performs comparably to or better than MEGA-G and PEG-G in some tasks.
- Ablations without DAD show only a small gap in Block Rotation and Pen Rotation, suggesting bidirectional transitions alone drive much of the gain there.
Limitations
- DAD may select ineffective or task-irrelevant subgoals in tasks with complex action spaces or weak correlation between goal and action spaces.
- Goal selection lacks a filter for infeasible goals, which can hurt sample efficiency (e.g., Fetch Slide).
- Only evaluated within Dreamer-based model-based RL; applicability to model-free RL is untested.
Experimental setup
Datasets: environment rollouts collected in replay buffer
Benchmarks: 3-Block Stacking Walker Ant Maze Block Rotation Pen Rotation Fetch Slide
Metrics: success rate one-step world model prediction error compounding error
Baselines: GC-Dreamer PEG-G MEGA-G MUN-noDAD MUN-KeyObs MUN-Ns-3
Builds on: Dreamer LEXA Go-Explore Plan2Explore PEG
Closest papers by meaning
- General agents need world models (ICML 2025 · similarity 0.88)
- RLVR-World: Training World Models with Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- PWM: Policy Learning with Multi-Task World Models (ICLR 2025 · similarity 0.88)
- Inverting the Bellman Equation: From $Q$-Values to World Models (NeurIPS 2026 · similarity 0.88)
- Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.