Research map / Vision-language and multimodal / Robot action and navigation
General Scene Adaptation for Vision-and-Language Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GSA-VLN, a persistent-environment VLN task, the GSA-R2R dataset (150 HM3D/MP3D buildings, multi-style instructions), and GR-DUET, a memory-graph method that reaches SOTA on GSA-R2R splits.
Problem
Standard VLN evaluates each instruction independently in unseen environments, ignoring that real-world robots operate persistently in consistent scenes and with consistent instructor styles; existing datasets also lack diverse, OOD evaluation scenes and instruction styles.
Approach
Frame navigation as continual single-scene adaptation with a persistent memory bank, build a benchmark with ID/OOD residential and non-residential buildings and LLM-based speaking-style instruction rephrasing, and propose GR-DUET, which keeps a global topological graph across episodes with environment-specific fine-tuning.</care_idea>
Key results
- GR-DUET reaches 69.3% SR on Test-R-Basic and 56.6% SR on Test-N-Basic versus 57.7% and 48.1% for DUET, with BT also improving some splits.
- TENT and SAR, entropy-based TTA, underperform vanilla DUET on Basic instructions, while memory-based TourHAMT and OVER-NAV drop far below HAMT/DUET.
- GR-DUET improves SR on Test-N-Scene from 40% (DUET) to 48.1%, and reaches 66.7% SR on Test-R-User for the Keith split.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| retain a global topological graph across episodes | DUET · GSA-R2R Test-R-Basic | SR: 57.7 -> 69.3 | Helped |
| add back-translation adaptation | DUET · GSA-R2R Test-R-Basic and Test-N-Basic | SR: 57.7 -> 61.3; 48.1 -> 49.5 | Helped |
| use entropy-based TENT or SAR adaptation | DUET · GSA-R2R Basic instructions | SR: 57.7 -> 57.2; 57.7 -> 57.6; 48.1 -> 46.5; 48.1 -> 44.6 | Hurt |
| add masked region classification (MRC) proxy-task adaptation | DUET · GSA-R2R Basic instructions | SR: 57.7 -> 57.7; 48.1 -> 48.1 | No clear effect |
| add PREVALENT augmentation without full-graph pretraining (ablation) | GR-DUET · Test-R-Basic, Test-N-Basic, Test-N-Scene | SR: 56.8 -> 54.0; 45.4 -> 43.0; 38.2 -> 35.9 | Hurt |
| add full-graph pretraining without PREVALENT augmentation (ablation) | GR-DUET · Test-R-Basic, Test-N-Basic, Test-N-Scene | SR: 56.8 -> 59.9; 45.4 -> 47.9; 38.2 -> 43.7 | Helped |
| combine full-graph pretraining with PREVALENT augmentation (ablation) | GR-DUET · Test-R-Basic, Test-N-Basic, Test-N-Scene | SR: 56.8 -> 69.3; 45.4 -> 56.6; 38.2 -> 48.1 | Helped |
| use a graph buffer of 1 episode instead of 50 (ablation) | GR-DUET · Test-R-Basic, Test-N-Basic, Test-N-Scene | SR: 69.3 -> 57.6; 56.6 -> 45.1; 48.1 -> 36.9 | Hurt |
| increase graph buffer from 50 to 100 episodes (ablation) | GR-DUET · Test-R-Basic, Test-N-Basic, Test-N-Scene | SR: 69.3 -> 69.7; 56.6 -> 56.1; 48.1 -> 47.9 | Mixed |
| refine speaker-generated instructions with Stage 2 (ablation) | Stage 1 instructions evaluated by DUET · Val-N-Scene, Val-R-User, Val-N-User | SR: 39.2 -> 42.8; 56.4 -> 59.1; 43.9 -> 46.4 | Helped |
| rephrase refined instructions into Scene and User speaking styles with Stage 3 (ablation) | Stage 2 instructions evaluated by DUET · Val-N-Scene, Val-R-User, Val-N-User | SR: 42.8 -> 39.3; 59.1 -> 55.6; 46.4 -> 44.1 | Hurt |
Surprising findings
- Entropy-based test-time adaptation methods (TENT, SAR) degrade performance relative to vanilla DUET on Basic and User instructions.
- Memory-based methods TourHAMT and OVER-NAV perform far worse than their base models when the number of stored instructions grows, due to excessively long history inputs.
Limitations
- Adaptation relies on unsupervised updates and the paper notes unsupervised approaches are ineffective in its setting.
- Instructions for Scene and User styles rely on GPT-4o generation, with human study covering only 20 instructions from 15 participants.
- ScaleVLN is evaluated with data leakage since it uses HM3D as augmented training data.
Experimental setup
Datasets: R2R PREVALENT HM3D MP3D SummScreen OpenStreetMap building tags ScaleVLN navigation graphs
Benchmarks: GSA-R2R R2R
Metrics: SR SPL NE TL nDTW
Baselines: HAMT DUET BEVBert ScaleVLN NavGPT-2 TENT SAR Back-Translation MLM MRC TourHAMT OVER-NAV
Builds on: DUET HAMT EnvDrop speaker RoleLLM TENT
Closest papers by meaning
- Towards Open Environments and Instructions: General Vision-Language Navigation via Fast-Slow Interactive Reasoning (CVPR 2026 · similarity 0.91)
- Iterative Vision-and-Language Navigation (CVPR 2023 · similarity 0.91)
- A New Path: Scaling Vision-and-Language Navigation With Synthetic Instructions and Imitation Learning (CVPR 2023 · similarity 0.91)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.88)
- Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions (NeurIPS 2024 · similarity 0.88)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.