Research map / Vision-language and multimodal / Robot action and navigation
Pretraining and data augmentation for VLN
Improves navigation agents through pretraining, synthetic instructions, imitation learning and exploration with scene objects.
97 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Robot action and navigation, keyword cluster “language navigation · vln · agent”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 5 of 6 tracked benchmarks.
18 papers in 2023–24 and 51 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 23.7% to 10.3%.
Open questions the data points to
- R2R val-unseen (SR): best went from 72 (up to 2024) to 84.4 in 2025 — Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel (ICLR 2025 · 0 citations)
- R2R Val Unseen (SPL): best went from 64.3 (up to 2024) to 65 in 2025 — Test-Time Adaptation for Online Vision-Language Navigation with Feedback-based Reinforcement Learning (ICML 2025)
- HM3D (SPL): best went from 26 (up to 2024) to 30.6 in 2025 — BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation (NeurIPS 2025 · 3 citations)
- REVERIE Val Unseen (SR): best went from 47.49 (up to 2024) to 66.49 in 2025 — Test-Time Adaptation for Online Vision-Language Navigation with Feedback-based Reinforcement Learning (ICML 2025)
- REVERIE val unseen (SPL): best went from 34.03 (up to 2024) to 45.4 in 2025 — Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel (ICLR 2025 · 0 citations)
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (CVPR 2018 · 1,195 citations)
- 2019 · Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (CVPR 2019 · 548 citations)
- 2020 · REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments (CVPR 2020 · 317 citations)
- 2021 · Topological Planning With Transformers for Vision-and-Language Navigation (CVPR 2021 · 106 citations)
- 2022 · Think Global, Act Local: Dual-Scale Graph Transformer for Vision-and-Language Navigation (CVPR 2022 · 176 citations)
- 2023 · A New Path: Scaling Vision-and-Language Navigation With Synthetic Instructions and Imitation Learning (CVPR 2023 · 35 citations)
- 2024 · NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · 184 citations)
- 2025 · Affordances-Oriented Planning Using Foundation Models for Continuous Vision-Language Navigation (AAAI 2025 · 21 citations)
- 2026 · MonoDream: Monocular Vision-Language Navigation with Panoramic Dreaming (AAAI 2026 · 2 citations)
Earliest papers here
- Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (CVPR 2018 · 1,195 citations)
- Speaker-Follower Models for Vision-and-Language Navigation (NeurIPS 2018 · 242 citations)
- Gated-Attention Architectures for Task-Oriented Language Grounding (AAAI 2018 · 125 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
R2R val-unseen · SR (higher is better)
9 papers, 2018–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 54.6 | Speaker-Follower Models for Vision-and-Language Navigation (NeurIPS 2018 · 242 citations) |
| 2020 | 56 | Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation (NeurIPS 2020 · 14 citations) |
| 2021 | 59 | SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation (NeurIPS 2021 · 4 citations) |
| 2022 | 72 | Think Global, Act Local: Dual-Scale Graph Transformer for Vision-and-Language Navigation (CVPR 2022 · 176 citations) |
| 2025 | 84.4 | Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel (ICLR 2025 · 0 citations) |
R2R Val Unseen · SPL (higher is better)
6 papers, 2020–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 44 | Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation (NeurIPS 2020 · 14 citations) |
| 2021 | 53 | SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation (NeurIPS 2021 · 4 citations) |
| 2023 | 64.3 | PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation (NeurIPS 2023 · 18 citations) |
| 2025 | 65 | Test-Time Adaptation for Online Vision-Language Navigation with Feedback-based Reinforcement Learning (ICML 2025) |
R2R test-unseen · SR (higher is better)
5 papers, 2018–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 53.5 | Speaker-Follower Models for Vision-and-Language Navigation (NeurIPS 2018 · 242 citations) |
| 2021 | 58 | SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation (NeurIPS 2021 · 4 citations) |
| 2022 | 69 | Think Global, Act Local: Dual-Scale Graph Transformer for Vision-and-Language Navigation (CVPR 2022 · 176 citations) |
| 2023 | 71 | Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding (CVPR 2023 · 21 citations) |
HM3D · SPL (higher is better)
4 papers, 2022–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 6.3 | Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation (CVPR 2022 · 55 citations) |
| 2023 | 22.3 | ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation (ICML 2023 · 22 citations) |
| 2024 | 26 | VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Model (ICML 2024 · 8 citations) |
| 2025 | 30.6 | BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation (NeurIPS 2025 · 3 citations) |
REVERIE Val Unseen · SR (higher is better)
4 papers, 2021–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 19.14 | Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (CVPR 2021 · 82 citations) |
| 2022 | 46.98 | Think Global, Act Local: Dual-Scale Graph Transformer for Vision-and-Language Navigation (CVPR 2022 · 176 citations) |
| 2023 | 47.49 | Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding (CVPR 2023 · 21 citations) |
| 2025 | 66.49 | Test-Time Adaptation for Online Vision-Language Navigation with Feedback-based Reinforcement Learning (ICML 2025) |
REVERIE val unseen · SPL (higher is better)
4 papers, 2021–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 11.84 | Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (CVPR 2021 · 82 citations) |
| 2022 | 33.73 | Think Global, Act Local: Dual-Scale Graph Transformer for Vision-and-Language Navigation (CVPR 2022 · 176 citations) |
| 2023 | 34.03 | Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding (CVPR 2023 · 21 citations) |
| 2025 | 45.4 | Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel (ICLR 2025 · 0 citations) |
What has been tried
57 papers here report 633 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 4%, hurt 19%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use FoV 45 with 24 views instead of FoV 60 with 12 views | Sampled 216 R2R trajectories · SPL | 6.52 -> 13.00 | Helped | NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · 184 citations) |
| Add BACL to GOAT without BACL or FACL | R2R validation unseen · SR | 75.27 -> 76.37 | Helped | Vision-and-Language Navigation via Causal Learning (CVPR 2024 · 40 citations) |
| Add instructions to VAP | R2R-CE validation unseen, 100-case ablation subset · SR | 24 -> 27; 3% improvement | Helped | Affordances-Oriented Planning Using Foundation Models for Continuous Vision-Language Navigation (AAAI 2025 · 21 citations) |
| add an explicit semantic and instance map | VAL SEEN · SPL | 9.4 -> 17.6 | Helped | GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation (CVPR 2024 · 18 citations) |
| restrict keywords to 12 common object categories instead of all open-vocabulary keywords | IR2R Val-Seen and Val-Unseen · t-nDTW | 62 ±0 -> 61 ±1; 55 ±1 -> 52 ±1 | Hurt | OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representation (CVPR 2024 · 13 citations) |
| Add Perception multimodal CoT to Perception + Decision | HM3D_v0.2 · SR | 0.677 -> 0.698 | Helped | Enhancing Multi-Robot Semantic Navigation Through Multimodal Chain-of-Thought Score Collaboration (AAAI 2025 · 9 citations) |
| add semantic guidance to Voronoi navigation | HM3D and HSSD · Success | HM3D: 38.7 -> 42.0; HSSD: 40.3 -> 41.0 | Helped | VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Model (ICML 2024 · 8 citations) |
| Add ImaAssistant to RILA | SoundSpaces Matterport3D test · SR (%) | 31.4 -> 35.4 | Helped | RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation (CVPR 2024 · 6 citations) |
| set imagination attention masks to zero at test time | R2R val-unseen · SR | 67.26 -> 66.92 | Hurt | Do Visual Imaginations Improve Vision-and-Language Navigation Agents? (CVPR 2025 · 5 citations) |
| use fusion encoder depth 3 instead of depth 1 | WAY valUnseen · Acc5 | 33.27 -> 38.23 | Helped | DiaLoc: An Iterative Approach to Embodied Dialog Localization (CVPR 2024 · 5 citations) |
| remove the belief-based planner | HM3D validation, 400 episodes · SR | 62.5 -> 56.0 | Hurt | BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation (NeurIPS 2025 · 3 citations) |
| Add visual neighbors' HTML text to the ranker | Mind2Web · Recall@1 | 25.4 -> 37.3 | Helped | Dual-View Visual Contextualization for Web Navigation (CVPR 2024 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 19 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- R2R-CE · 5 of 19 papers
- R2R · 4 of 19 papers
- RxR-CE · 3 of 19 papers
- MP3D · 3 of 19 papers
- REVERIE · 3 of 19 papers
- Matterport3D · 3 of 19 papers
- HM3D · 3 of 19 papers
- FG-R2R · 2 of 19 papers
Baselines
- ETPNav · 4 of 19 papers
- BEVBert · 4 of 19 papers
- Uni-NaVid · 3 of 19 papers
- ZSON · 2 of 19 papers
- Cow · 2 of 19 papers
- NaVid · 2 of 19 papers
- L3MVN · 2 of 19 papers
- ESC · 2 of 19 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 69 papers; from OpenAlex and the papers' affiliations).
- The University of Adelaide · 3 papers
- The University of Hong Kong · 3 papers
- Carnegie Mellon University · 3 papers
- Zhejiang University · 2 papers
- Google · 2 papers
- Apple · 2 papers
- University of Illinois, Urbana Champaign · 2 papers
- University of Washington · 2 papers
- Tongji University · 2 papers
- University of Michigan - Ann Arbor · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Reasoning-enhanced vision-language navigation (46 papers)
Elsewhere in the topic
- Vision-language-action foundation models (264 papers)
- Memory and subgoal-based manipulation (103 papers)
- Embodied multimodal LLM agents (70 papers)
- Multimodal prompt-driven manipulation (50 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).