Research map / Vision-language and multimodal / Robot action and navigation
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NavGPT, a zero-shot LLM-based vision-and-language navigation agent that converts visual observations to text via VFMs and performs explicit ReAct-style reasoning with GPT-4 and GPT-3.5.
Problem
Vision-and-language navigation (VLN) requires agents to follow natural language instructions in unseen environments; the reasoning ability of LLMs for navigation decision-making is under-explored, and zero-shot LLM agents lag far behind trained models.
Approach
Treat visual observations as text (BLIP-2 captions, Faster R-CNN object detections with depth, GPT-3.5 summaries) and let a frozen GPT-4 interleave explicit reasoning thoughts with navigation actions over a summarized history, without any VLN training.
Key results
- On R2R val unseen (783 trajectories), NavGPT with GPT-4 reaches 34% SR and 29 SPL zero-shot, versus 72% SR for DuET trained with pretraining and finetuning.
- With GPT-3.5 in ablations, adding object detections raised SR from 11.11 to 15.97, and adding depth further raised SR to 16.67 and SPL to 13.00 (FoV@45, 24 views setting).
- GPT-4 generated plausible navigation instructions from trajectories and drew top-down trajectory maps, showing spatial and historical awareness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FoV 45 with 24 views instead of FoV 60 with 12 views (ablation) | NavGPT visual observation descriptions · Sampled 216 R2R trajectories | SPL: 6.52 -> 13.00 | Helped |
| Use FoV 45 with 24 views instead of FoV 30 with 36 views (ablation) | NavGPT visual observation descriptions · Sampled 216 R2R trajectories | SPL: 9.12 -> 13.00 | Helped |
| Add object information (ablation) | NavGPT with GPT-3.5 · Sampled 216 R2R trajectories | SR: 11.11 -> 15.97 | Helped |
| Add depth information to object information (ablation) | NavGPT with GPT-3.5 · Sampled 216 R2R trajectories | SR: 15.97 -> 16.67 | Helped |
| Add depth information to object information (ablation) | NavGPT with GPT-3.5 · Sampled 216 R2R trajectories | SPL: 11.71 -> 13.00 | Helped |
Surprising findings
- Zero-shot NavGPT with GPT-4 is competitive with some earlier trained-only baselines (e.g., Seq2Seq at 21 SR) despite using no VLN training.
Limitations
- Performance gap to trained models due to information loss when translating visuals into text captions.
- Summarizing history into one sentence per viewpoint loses details, causing progress-tracking errors.
- Evaluation limited to a 783-trajectory subset of R2R val unseen and ablations on a 216-sample subset with GPT-3.5.
Experimental setup
Datasets: R2R
Benchmarks: R2R
Metrics: TL NE OSR SR SPL
Baselines: Seq2Seq Speaker Follower EnvDrop PREVALENT VLN↻BERT HAMT DuET DuET (Init. LXMERT)
Builds on: ReAct BLIP-2 Faster R-CNN Matterport3D simulator
Closest papers by meaning
- FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-and-Language Navigation (CVPR 2026 · similarity 0.92)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.92)
- Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation (NeurIPS 2025 · similarity 0.92)
- Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation (AAAI 2026 · similarity 0.91)
- Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.90)
- Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.