Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Towards Learning a Generalist Model for Embodied Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NaviLLM, a generalist embodied navigation model that adapts Vicuna-7B to multiple navigation and 3D QA tasks via schema-based instruction and multi-task training.
Problem
Task-specific embodied navigation agents generalize poorly to unseen tasks and scenarios; a single model is needed that handles diverse embodied navigation tasks.
Approach
Cast all embodied tasks (VLN, object localization, trajectory summarization, 3D-QA, EQA) into generation via schema-based instructions (task, observation, history, output hint) fed to a fine-tuned LLM with multi-view scene encoding.
Key results
- Achieves 7.90 GP on CVDN test versus 6.13 for prior SoTA VLN-PETL, a relative gain of about 29%.
- Reaches 26.3 EM on ScanQA test, surpassing 3D-LLM's 19.1 EM.
- In held-out SOON experiment, improves SPL by 136% over DUET (21.04 vs 10.91 on REVERIE-excluded setting is separate; SOON held-out SPL 51.47 vs 21.42-level DUET baseline per table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train jointly on diverse embodied tasks instead of without multi-task learning (ablation) | NaviLLM · CVDN, SOON, R2R, REVERIE, and ScanQA | performance: 5.54 -> 5.91 GP; 28.37 -> 35.44 SR; 21.8 -> 23.3 EM | Helped |
| initialize the LLM with pretrained weights instead of random initialization (ablation) | NaviLLM · CVDN, SOON, R2R, REVERIE, and ScanQA | performance: 3.64 -> 5.91 GP; 20.73 -> 35.44 SR; 12.4 -> 23.3 EM | Helped |
| pretrain on augmented R2R and REVERIE data (ablation) | NaviLLM · CVDN, SOON, and R2R | performance: 5.91 -> 6.16 GP; 35.44 -> 38.33 SR; 44.56 -> 42.15 SR | Mixed |
| use NaviLLM rather than VLN-PETL | CVDN · CVDN test | SPL: 6.13 -> 7.90 | Helped |
| use NaviLLM rather than Meta-Explore | SOON · SOON test | SPL: 25.80 -> 26.26 | Helped |
| use NaviLLM rather than Meta-Explore | R2R · R2R test | SPL: 61 -> 60 | Hurt |
| use NaviLLM rather than DUET | REVERIE · REVERIE test | SPL: 36.06 -> 32.33 | Hurt |
| use NaviLLM rather than the strongest listed ScanQA baseline | ScanQA · ScanQA test | EM: 23.2 -> 26.3 | Helped |
Surprising findings
- Pre-training on augmented R2R/REVERIE data gave only slight gains on R2R, CVDN, and SOON, contrary to prior work.
- Model underperforms DUET on REVERIE, which has simpler instructions, suggesting the generalist approach trades off on simple-instruction tasks.
Limitations
- Lags state-of-the-art DUET on REVERIE with simpler instructions.
- Gains from augmented pre-training data are limited, suggesting data quality matters more than quantity.
Experimental setup
Datasets: CVDN SOON R2R REVERIE ScanQA LLaVA-23k MP3D-EQA
Benchmarks: CVDN SOON R2R REVERIE ScanQA MP3D-EQA
Metrics: SPL SR GP OSR TL EM ROUGE-L METEOR CIDER BLEU-4
Baselines: DUET HAMT AZHP BEV-BERT 3D-LLM ScanQA MT-RCM+Env VLN-PETL
Builds on: LLaVA DUET schema-based instruction for dialog
Closest papers by meaning
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.89)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.89)
- TANGO: Training-free Embodied AI Agents for Open-world Tasks (CVPR 2025 · similarity 0.89)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.88)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.88)
- VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.