Research map / Vision-language and multimodal / Robot action and navigation
NavBench: Probing Multimodal Large Language Models for Embodied Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NavBench, a zero-shot benchmark probing MLLMs for embodied navigation via three comprehension tasks (3,200 QA pairs) and difficulty-stratified step-by-step viewpoint-selection execution in Matterport3D (432 episodes, 72 scenes), plus a real-robot deployment pipeline.
Problem
Existing embodied navigation benchmarks mainly report final success rates for supervised agents and do not reveal whether MLLMs understand navigation behavior or can act step-by-step with difficulty-aware evaluation.
Approach
Decompose MLLM navigation evaluation into cognitively grounded comprehension tasks (global instruction alignment, temporal progress estimation, local observation-action reasoning) and a step-by-step execution task stratified into easy/medium/hard by a composite spatial, cognitive, and execution complexity score, then map MLLM outputs to robot actions.
Key results
- GPT-4o achieves the highest comprehension average (53.34%) and execution average (41.33%) among evaluated models.
- Qwen2.5-VL-7B is the best open-source model (45.26% comprehension, 21.77% execution average), approaching GPT-4o-mini (46.42%, 27.99%).
- Progress estimation is a consistent weakness: only GPT-4o exceeds 42.90% on it while most other models perform poorly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide map connectivity as text-prompt context | GPT-4o · Medium navigation | success rate (SR): +4.86 percentage points | Helped |
| use map connectivity as context | GPT-4o · Navigation Execution across difficulty levels | success rate (SR) | Helped |
| reorder sub-instructions while retaining their content (shuffle condition) | MLLMs on Global Instruction Alignment · Shuffled instruction perturbation | accuracy | Hurt |
| use GPT-4o rather than other evaluated models | Navigation Comprehension and Execution · NavBench | comprehension average and execution average: 53.34; 41.33 | Helped |
| use InternVL2.5-2B rather than GPT-4o | Global Instruction Alignment · Global Instruction Alignment | accuracy: 51.33 -> 67.25 | Helped |
| test models on real-world indoor navigation cases | GPT-4o · Real-world navigation | success rate: 60% | Mixed |
| test models on real-world indoor navigation cases | Qwen2.5-VL-7B · Real-world navigation | success rate: 40% | Mixed |
Surprising findings
- InternVL2.5-2B reaches 67.25% on global instruction alignment, surpassing GPT-4o (51.33%) despite weak progress and local scores.
- All models perform poorly on the shuffled-segment distractor condition, indicating weak temporal order reasoning in instructions.
Limitations
- Execution uses abstracted viewpoint selection rather than low-level control, and real-world validation is only 10 cases per model.
- Difficulty weights alpha, beta, gamma are described as empirically set without reported values.
- Human reference computed on a small VLN-Bench (tiny) subset of 392 items.
Experimental setup
Datasets: R2R RxR GEL-R2R FGR2R Matterport3D
Benchmarks: NavBench
Metrics: accuracy SR SPL
Baselines: GPT-4o-mini InternVL2.5-2B InternVL2.5-8B Qwen2.5-VL-3B Qwen2.5-VL-7B LLaVA-OneVision-7B Llama3.2-Vision-11B Chance level (random)
Builds on: R2R RxR MapGPT Matterport3D
Closest papers by meaning
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.92)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.91)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.91)
- Scene Map-based Prompt Tuning for Navigation Instruction Generation (CVPR 2025 · similarity 0.90)
- RAO-Nav: Probing Omni-Language Models for Zero-shot Semantic Audio-Visual Navigation (NeurIPS 2026 · similarity 0.90)
- ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.