Research map / Reinforcement learning / Visual and representation RL
Leveraging Separated World Model for Exploration in Visually Distracted Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeeX, a bi-level model-based unsupervised RL method that trains a separated world model (endogenous/exogenous latents) and explores by maximizing endogenous ensemble-disagreement uncertainty, evaluated on distracted DMC tasks.
Problem
Distractors in visual observations corrupt intrinsic reward estimation in model-based unsupervised reinforcement learning, biasing exploration in pixel-based environments.
Approach
Separate the latent world model into task-relevant (endogenous) and task-irrelevant (exogenous) branches, compute intrinsic reward as ensemble variance only over the endogenous predictive heads, and train the policy solely on imaginary trajectories in the endogenous state space.
Key results
- SeeX reaches 0.95 normalized Jaco performance at 100K pre-training frames versus at most 0.54 for baselines (Plan2Explore 0.54, others below 0.5 at 100K).
- On OOD random-video distractors, SeeX scores 0.75 on Random-video versus 0.82 on Driving-car (drop of -0.09), the smallest drop among compared methods (e.g., APS -0.51).
- SeeX reaches 90% of expert performance on Walker-Walk within 500K pre-training frames, while no baseline reaches it even at 2M frames.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SeeX rather than LBS | Walker tasks with visual distractors · Random-video | average normalized return: 0.61± 0.08 -> 0.75± 0.07 | Helped |
| use SeeX rather than Plan2Explore | Walker · Walker Walk | normalized return: 0.77 -> 0.91 | Helped |
| add a separated world model (ablation) | SeeX · Quadruped, 500k pre-training frames | normalized return: 0.15 -> 0.49 | Helped |
| add a separated world model (ablation) | SeeX · Walker, 500k pre-training frames | normalized return: 0.65 -> 0.80 | Helped |
| add Exo-Rec (ablation) | SeeX · Quadruped, 500k pre-training frames | normalized return: 0.36 -> 0.49 | Helped |
| add Exo-Rec (ablation) | SeeX · Walker, 500k pre-training frames | normalized return: 0.69 -> 0.80 | Helped |
| increase predictive heads from K=5 to K=10 (ablation) | SeeX · Quadruped, 500k pre-training frames | normalized return: 0.49 -> 0.47 | No clear effect |
| increase predictive heads from K=5 to K=10 (ablation) | SeeX · Walker, 500k pre-training frames | normalized return: 0.80 -> 0.82 | Helped |
| increase predictive heads from K=5 to K=2 (ablation) | SeeX · Quadruped, 500k pre-training frames | normalized return: 0.49 -> 0.37 | Hurt |
| increase predictive heads from K=5 to K=2 (ablation) | SeeX · Walker, 500k pre-training frames | normalized return: 0.80 -> 0.71 | Hurt |
| use the endogenous state s+ alone for policy input instead of s+ and s− (ablation) | SeeX · Walker and Quadruped tasks | normalized return | Helped |
| apply data augmentation during fine-tuning (ablation) | SeeX · Walker tasks | normalized return | Mixed |
Surprising findings
- Policies using both endogenous and exogenous branches underperform the endogenous-only policy, suggesting exogenous information harms policy learning under moving distractors.
- Data augmentation during pre-training (pDA) reduced performance, which the authors attribute to disruption of world model learning.
Limitations
- Evaluated only on DMC-based tasks; real-world applications such as self-driving and navigation not tested.
- Restricted to task-irrelevant, action-independent distractors; other distractor types left to future work.
- Jaco reconstruction quality is worse and distractor removal less effective for complex multi-joint arms.
Experimental setup
Datasets: Kinetics (Driving-car subset) Kinetics (Random-video subset) DMC Walker DMC Quadruped Jaco Arm (URLB)
Benchmarks: URLB-pixels with distractors (Walker, Quadruped, Jaco) OOD random-video distractors
Metrics: normalized return uncertainty estimate (U(s+), U(s-), U(z))
Baselines: ICM RND LBS Plan2Explore APT DIAYN APS
Builds on: Dreamer Plan2Explore URLB-pixels EX-BMDP SeMAIL
Closest papers by meaning
- Learning World Models for Unconstrained Goal Navigation (NeurIPS 2024 · similarity 0.85)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.85)
- Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration (ICML 2025 · similarity 0.85)
- CIG: Exploration via Conditional Information Gain (NeurIPS 2026 · similarity 0.85)
- Open-World Reinforcement Learning over Long Short-Term Imagination (ICLR 2025 · similarity 0.85)
- MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.