Research map / Vision-language and multimodal / Robot action and navigation
Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Self-Refining Data Flywheel where an instruction generator (Mantis-based) and a DUET navigator iteratively filter and regenerate instruction-trajectory data for vision-and-language navigation, without human annotation.
Problem
Synthetic instruction-trajectory data for vision-and-language navigation (VLN) is of low language fidelity and alignment, and human annotation is costly, limiting navigator and generator quality.
Approach
A two-model loop: the navigator scores generated instructions by re-running them and comparing the followed path to the original (nDTW/SPL) to filter data for training a better generator, and the improved generator regenerates the low-quality samples to train a stronger navigator, iterated over several rounds.
Key results
- Navigator SPL on R2R test unseen rises to 85% SR and 78% SPL (vs. ScaleVLN 80% SR / 70% SPL), surpassing human 76% SPL for the first time.
- Instruction generator SPICE on R2R val unseen increases from 23.5 (prior SoTA) to 26.2 after a fourth round of generator training.
- Navigator-nDTW filtering yields SPICE 25.4 vs. 23.7 without filtering and 24.4 with CLIP-Sim filtering.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the SRDF-trained navigator instead of ScaleVLN | DUET on R2R test unseen · R2R test unseen | SPL: 70 -> 78 | Helped |
| Use the SRDF-trained navigator instead of BEVBert | DUET on R2R-CE test unseen · R2R-CE test unseen | SPL: 50 -> 56 | Helped |
| Use navigator-filtered data to train the next-round generator | SRDF instruction generator · R2R val unseen | SPICE: 23.7 -> 25.2 | Helped |
| Refine navigation instructions with the improved generator for another round | SRDF navigator · R2R val unseen | SR: +1.2% SR in round 2, and +0.8% SR in round 3 | Helped |
| Filter instructions with navigator-nDTW instead of no filter (ablation) | SRDF instruction generator · R2R val unseen | SPICE: 23.7 -> 25.4 | Helped |
| Filter instructions with CLIP-Sim instead of no filter (ablation) | SRDF instruction generator · R2R val unseen | SPICE-D: 28.4 -> 28.7 | Helped |
| Filter instructions with Mantis-Score instead of no filter (ablation) | SRDF instruction generator · R2R val unseen | SPICE: 23.7 -> 23.6 | Hurt |
| Increase generated instructions per path from 1 to 6 (ablation) | SRDF navigator training · R2R val unseen | SR: 73.86 -> 76.93 | Helped |
| Increase generated instructions per path from 6 to 12 (ablation) | SRDF navigator training · R2R val unseen | SPL: 66.89 -> 66.53 | No clear effect |
| Add ScaleVLN data to generator training (ablation) | Round-1 SRDF instruction generator · R2R val unseen | SPICE: 23.7 -> 23.5 | Hurt |
| Add Prevalent data to generator training (ablation) | Round-1 SRDF instruction generator · R2R val unseen | SPICE: 23.7 -> 23.6 | Hurt |
| Add high-quality generated data from 800 HM3D environments (ablation) | Round-1 SRDF instruction generator · R2R val unseen | SPICE: 23.7 -> 25.2 | Helped |
Surprising findings
- Training solely on the large synthetic datasets (ScaleVLN, Prevalent) underperforms training on the small human-labeled R2R set in SPL.
- Self-scoring by the language model and CLIP-score filtering give no significant gain for generator training, while navigator-based filtering does.
Limitations
- Gains depend on the navigator being strong enough to act as a reliable filter; the paper notes performance bounded by navigator quality.
- Experiments are primarily on R2R and downstream tasks use the R2R-style DUET architecture, limiting evidence of generality across navigator architectures.
Experimental setup
Datasets: R2R SRDF-20M ScaleVLN Prevalent RxR-English CVDN REVERIE SOON R4R R2R-CE MP3D HM3D
Benchmarks: R2R R2R-CE RxR-English R4R CVDN REVERIE SOON
Metrics: SR SPL nDTW sDTW GP NE OSR SPICE SPICE-D BLEU
Baselines: ScaleVLN ETPNav BEVBert GOAT MAGIC-L EnvDrop KEFA CLIP-Sim Mantis-Score AutoVLN
Builds on: DUET Mantis ScaleVLN Speaker-Follower PanoGen
Closest papers by meaning
- Scene Map-based Prompt Tuning for Navigation Instruction Generation (CVPR 2025 · similarity 0.86)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.86)
- Towards Learning a Generalist Model for Embodied Navigation (CVPR 2024 · similarity 0.86)
- STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimization (NeurIPS 2025 · similarity 0.85)
- General Scene Adaptation for Vision-and-Language Navigation (ICLR 2025 · similarity 0.85)
- Self-Improving Embodied Foundation Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.