Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SALI, a VLN agent with a reality-imagination hybrid topological memory in which a recurrent imagination tree generates future depth, semantic, and RGB views that are fused into navigation decisions.
Problem
Vision-and-language navigation agents generalize poorly to unseen environments and lack a persistent memory that integrates imagined future scenes over time.
Approach
A recurrent, tree-structured imagination module (inpaint, SPADE, room-type and waypoint models) generates imagined nodes that are pruned and merged into a topological memory alongside real nodes, with a dynamic fusion factor that weights imagined navigation scores.
Key results
- On R2R Test Unseen, SALI reaches SR 79 and SPL 74, versus 77 and 68 for ScaleVLN.
- On REVERIE Test Unseen, SALI reaches RGS 34 and RGSPL 25, the best among compared methods.
- Ablation on R2R Val Unseen: reality plus imagination memory gives SR 82 and SPL 70 versus 70 and 61 for reality only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reality + imagination memory instead of reality-only memory (ablation) | SALI · R2R Val Unseen | SR: 70 -> 82 | Helped |
| Use imagination-only memory instead of reality + imagination memory (ablation) | SALI · R2R Val Unseen | SR: 82 -> 58 | Hurt |
| Use reality + imagination memory instead of reality-only memory (ablation) | SALI · R2R Val Unseen | SPL: 61 -> 70 | Helped |
| Expand imagination range from M=0, N̄=0 to M=2, N̄=4 (ablation) | SALI · R2R Val Unseen | SPL: 61 -> 71 | Helped |
| Expand imagination node limit from N̄=4 to N̄=8 at M=2 (ablation) | SALI · R2R Val Unseen | SPL: 71 -> 68 | Hurt |
| Add the room-type model to the no-auxiliary-model variant (ablation) | SALI · R2R Val Unseen | SPL: 61 -> 64 | Helped |
| Add the waypoint model to the no-auxiliary-model variant (ablation) | SALI · R2R Val Unseen | SPL: 61 -> 66 | Helped |
| Add both room-type and waypoint models (ablation) | SALI · R2R Val Unseen | SPL: 61 -> 71 | Helped |
| Use dynamic rather than fixed γt=0.5 weighting (ablation) | SALI · R2R Val Unseen | SPL: 67 -> 71 | Helped |
| Use SALI rather than Lily | R2R · Val Unseen | SPL: 72 -> 78 | Helped |
| Use SALI rather than RecBert | R2R · Test Unseen | SPL: 57 -> 74 | Helped |
| Use SALI rather than BEVBert | REVERIE · Val Unseen | RGSPL: 24 -> 28 | Helped |
Surprising findings
- Imagination-only navigation is the least effective memory type (SR 58), below reality-only (SR 70).
- Enlarging the imagination range to M=2, N=8 lowers SPL relative to M=2, N=4.
Limitations
- Computational cost of the imagination process is a stated concern; future work targets efficiency.
- Imagination-only navigation performs poorly, indicating reliance on real observations.
Experimental setup
Datasets: R2R REVERIE Matterport3D Simulator
Benchmarks: R2R REVERIE
Metrics: NE OSR SR SPL RGS RGSPL TL PSNR
Baselines: Seq2Seq RecBert HOP+ DUET BEVBert Lily ScaleVLN
Builds on: LXMERT BEVBert SPADE DUET
Closest papers by meaning
- Do Visual Imaginations Improve Vision-and-Language Navigation Agents? (CVPR 2025 · similarity 0.91)
- ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination (ICLR 2025 · similarity 0.90)
- Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators (NeurIPS 2026 · similarity 0.88)
- Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.87)
- RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation (CVPR 2024 · similarity 0.87)
- FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-and-Language Navigation (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.