Research map / Reinforcement learning / Visual and representation RL
SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SENSEI, which distills a VLM-derived interestingness reward (VLM-Motif) into a world model and uses model-based RL (DreamerV3) to jointly maximize semantic reward and ensemble disagreement for semantically meaningful task-free exploration.
Problem
Intrinsic-motivation exploration based on novelty or information gain tends to uncover only low-level interactions, and existing foundation-model-guided exploration methods assume language-grounded environments, offline exhaustive datasets, or high-level actions.
Approach
Distill a VLM's pairwise interestingness preferences over images into a reward model (VLM-Motif), have the RSSM world model predict this semantic reward as an extra head, and train an exploration policy with a Go-Explore-style adaptive switch between semantic reward and ensemble disagreement.
Key results
- In MiniHack KeyRoom, SENSEI-based downstream learning solves the task roughly two orders of magnitude faster than PPO (PPO needs >20M steps).
- In Pokémon Red, only SENSEI reaches the first Gym (segment 9) within 750k exploration steps, while Plan2Explore and DreamerV3 do not.
- On Robodesk, SENSEI interacts with most objects more than Plan2Explore and RND during 1M steps of exploration, and a zero-knowledge prompt variant performs comparably.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine semantic reward with disagreement/information-gain reward rather than using semantic reward alone (ablation) | SENSEI on MiniHack · KeyRoom-S15 and KeyChest | task-relevant interactions and task rewards | Helped |
| Use VLM-generated environment description instead of a human-provided description (ablation) | SENSEI on Robodesk · Robodesk | object interactions | No clear effect |
| Use semantic VLM guidance and disagreement-based exploration instead of Plan2Explore | SENSEI · MiniHack KeyRoom-S15 and KeyChest | task-relevant interactions | Helped |
| Use semantic VLM guidance and disagreement-based exploration instead of Plan2Explore or RND | SENSEI · Robodesk | object interactions | Helped |
| Initialize downstream DreamerV3 from a SENSEI exploration world model instead of training from scratch or using the Plan2Explore world model | DreamerV3 · MiniHack KeyRoom-S15 and KeyChest | episode score: roughly two orders of magnitude | Helped |
| Use SENSEI exploration with extrinsic rewards instead of Plan2Explore or extrinsic rewards alone | SENSEI general · Pokémon Red | map coverage and progress to first Gym | Helped |
| Use SENSEI exploration with extrinsic rewards instead of Plan2Explore | SENSEI general · Pokémon Red | highest Pokémon level: twice as many level-ups per episode | Helped |
| Refine the semantic reward using a second round of VLM annotations on SENSEI-collected data (ablation) | SENSEI · Pokémon Red | Gym progress: 200 additional episodes | Helped |
| Replace VLM annotations with oracle annotations (ablation) | SENSEI on Robodesk · Robodesk | object interactions | Helped |
| Use a richer CEE-US initial exploration dataset instead of Plan2Explore data (ablation) | SENSEI on Robodesk · Robodesk | object interactions | Helped |
| Replace Go-Explore switching with fixed weights for semantic and disagreement rewards (ablation) | SENSEI on Robodesk · Robodesk | hyperparameter sensitivity | Mixed |
Surprising findings
- Pure semantic reward without information gain (VLM-Motif alone) gets stuck in local optima, e.g. picking up the key but not opening the door or chest.
- Injecting a human-written environment description into the VLM prompt is not necessary; a VLM-generated description performs about as well on Robodesk.
Limitations
- Requires fully observable image observations; VLM annotations degrade under occlusion.
- Semantic reward degrades out of the annotation data distribution in the large Pokémon Red world, requiring a second annotation round.
- Experiments are limited to simulated environments (MiniHack, Robodesk, Pokémon Red).
Experimental setup
Datasets: Plan2Explore exploration data (initial D_init) VLM-annotated preference pairs
Benchmarks: MiniHack KeyRoom-S15 MiniHack KeyChest Robodesk Pokémon Red
Metrics: mean object interactions task rewards collected during exploration downstream episode score map segments visited highest Pokémon level
Baselines: Plan2Explore Random Network Distillation DreamerV3 PPO VLM-Motif (pure semantic reward)
Builds on: DreamerV3 Plan2Explore MOTIF Go-Explore RL-VLM-F
Closest papers by meaning
- Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data (NeurIPS 2025 · similarity 0.87)
- SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.87)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.87)
- Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.87)
- A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.