Research map / Vision-language and multimodal / Robot action and navigation
ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ImagineNav++, a training-free mapless navigation framework that imagines future views (ResNet-18 waypoint predictor plus diffusion NVS model Polyoculus) and uses a VLM to pick the best view, with a keyframe-based selective foveation memory.
Problem
Open-vocabulary goal-oriented visual navigation (ObjectNav and instance-image-goal navigation) for home-assistant robots; existing LLM-based pipelines rely on maps and text-only representations that cannot capture spatial occupancy or geometry.
Approach
Recast navigation as best-view image selection: a Where2Imagine module predicts candidate relative waypoint poses from human demonstrations, a novel view synthesis model renders imagined views, and a VLM selects the most promising view, with DINOv2-based keyframe memory providing long-term context and PointNav executing the move.
Key results
- ImagineNav++ reaches 58.5% SR and 26.6% SPL on HM3D ObjectNav and 64.5% SR on HSSD, exceeding the strongest listed open-vocabulary baselines in SR there.
- On HM3D instance-image-goal navigation, ImagineNav++ achieves 52.4% SR and 32.8% SPL, the highest SPL among compared methods including map-based ones.
- Qwen3-VL-32B and Qwen3.5-27B reach 71.0% SR on HM3D ObjectNav, slightly above GPT-4o-mini (70.0%) in the reported ablation setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ImagineNav++ instead of the strongest listed baseline | ObjectNav · HM3D | SR: 54.5 -> 58.5 | Helped |
| use ImagineNav++ instead of the strongest listed baseline | ObjectNav · HSSD | SR: 51.0 -> 64.5 | Helped |
| use ImagineNav++ instead of the strongest listed baseline | InsINav · HM3D | SPL: 24.1 -> 32.8 | Helped |
| replace synthesized candidate images with real candidate-point images | ImagineNav++ · Gibson ObjectNav | SR: 72.4 -> 77.1 | Helped |
| add future-view imagination (ablation) | ImagineNav++ without Where2Imagine or memory · ObjectNav | SR: 43.0 -> 55.0 | Helped |
| add future-view imagination (ablation) | ImagineNav++ without Where2Imagine or memory · InsINav | SR: 34.9 -> 47.2 | Helped |
| add Where2Imagine to imagination with Oracle views (ablation) | ImagineNav++ without memory · ObjectNav | SR: 55.0 -> 64.0 | Helped |
| add Where2Imagine to imagination with Oracle views (ablation) | ImagineNav++ without memory · InsINav | SR: 47.2 -> 50.3 | Helped |
| add memory with PolyOculus NVS (ablation) | ImagineNav++ with Where2Imagine · ObjectNav | SR: 56.0 -> 67.0 | Helped |
| add memory with PolyOculus NVS (ablation) | ImagineNav++ with Where2Imagine · InsINav | SR: 43.4 -> 48.8 | Helped |
| use real views instead of PolyOculus-synthesized views (ablation) | ImagineNav++ without memory · ObjectNav | SR: 56.0 -> 64.0 | Helped |
| use real views instead of PolyOculus-synthesized views (ablation) | ImagineNav++ with memory · ObjectNav | SR: 67.0 -> 70.0 | Helped |
Surprising findings
- Full (non-keyframe) memory did not improve SR over no-memory baseline (56.0 vs 56.0), though it improved SPL slightly.
- Locally deployable open-source Qwen models matched or slightly exceeded GPT-4o-mini performance.
Limitations
- Performance limited by quality of off-the-shelf NVS model; synthesized views can contain artifacts or non-existent objects.
- Relies on closed-source GPT API for main results; latency from VLM calls.
- Simulator annotation errors cause false failures in evaluation.
Experimental setup
Datasets: Habitat-Web human demonstrations (MP3D) Gibson HM3D HSSD
Benchmarks: Gibson ObjectNav HM3D ObjectNav HSSD ObjectNav HM3D InsINav
Metrics: SR SPL
Baselines: FBE SemExp Habitat-Web OVRL ESC VoroNav VLFM Goat SG-Nav UniGoal ZSON PixNav
Builds on: VLFM NaVid DINOv2 VER Habitat-Web
Closest papers by meaning
- Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigation (AAAI 2025 · similarity 0.90)
- FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-and-Language Navigation (CVPR 2026 · similarity 0.89)
- Do Visual Imaginations Improve Vision-and-Language Navigation Agents? (CVPR 2025 · similarity 0.88)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.88)
- Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.88)
- Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.