Research map / Video / Video-language understanding
VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Augments outdoor VLN pretraining with template-infilled navigation instructions and rotation-similarity action labels derived from BDD100K driving videos, achieving new state-of-the-art on Touchdown.
Problem
Outdoor vision-and-language navigation suffers from limited training data and limited environment diversity, and existing speaker-based instruction augmentation does not transfer well to outdoor driving videos.
Approach
Generate VLN-style data from caption-less driving videos by template infilling (noun phrases masked and refilled with detected objects, directions filled by an image rotation similarity action predictor), then pre-train a VLN-Transformer with MLM, instruction-trajectory matching, and next action prediction before transferring the instruction encoder to the ORAR navigator.
Key results
- On the Touchdown test set, VLN-Video reaches 31.7 TC, improving over ORAR-BERT (30.0) by 1.7 points and claiming SOTA among methods without ground-truth path traces.
- Template infilling outperforms the learned speaker by 3.5 TC on validation (ablation '-speaker').
- Rotation-similarity action predictor outperforms the learned turn predictor by 3.2 TC on validation; human check showed the learned turn predictor failed on all turns.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train with Touchdown and processed BDD100K driving videos | ORAR-BERT · Touchdown test | TC: 2.1% | Helped |
| Pre-train on Touchdown and driving videos instead of ORAR-BERT alone | ORAR-BERT · Touchdown validation | TC: 3.9% | Helped |
| Pre-train on Touchdown only (ablation) | ORAR-BERT · Touchdown validation | TC: 1.0% | Helped |
| Add original Manh50 instructions to Touchdown pre-training (ablation) | ORAR-BERT · Touchdown validation | TC: 1.6% | Helped |
| Use VLN-Video-generated instructions rather than original or style-transferred Manh50 instructions (ablation) | ORAR-BERT pre-trained with Touchdown and Manh50 · Touchdown validation | TC: more than 2% | Helped |
| Use style-transferred Manh50 instructions (ablation) | ORAR-BERT pre-trained with Touchdown and Manh50 · Touchdown test | TC | Mixed |
| Replace template infilling with a learned LSTM speaker (ablation) | VLN-Video · Touchdown validation | TC: 3.5% | Hurt |
| Remove GPT-2 template filtering (ablation) | VLN-Video · Touchdown validation | TC: 2.9% | Hurt |
| Replace rotation-similarity action prediction with a learned turn predictor (ablation) | VLN-Video · Touchdown validation | TC: 3.2% | Hurt |
| Use attended 36 panorama views instead of the middle panorama views (ablation) | ORAR · Touchdown validation | TC: 10.8% | Hurt |
| Add visual view features to action representations (ablation) | 36-view panorama agent · Touchdown validation | TC: 9.1% | Helped |
| Drop half the panorama views and retain the 18 front views (ablation) | 36-view panorama agent · Touchdown validation | TC: 2.6% | Helped |
Surprising findings
- A learned turn predictor trained on Touchdown predicted forward for nearly all driving-video frame pairs due to domain gap, so a simple rotation-similarity heuristic was used instead.
- PM-VLN, which uses ground-truth path traces, still has higher test TC (33.4) than VLN-Video (31.7).
Limitations
- Turn prediction from driving videos is weakly accurate (inter-annotator agreement 0.493; 33% of turns correctly predicted in human check).
- Performance gains are modest (2.1% TC over prior SOTA) and the test set improvement is smaller than the validation improvement.
- Uses only 6K clear-daytime BDD100K videos from a few cities, and relies on manually filtered object classes.
Experimental setup
Datasets: Touchdown Manh50 BDD100K LVIS StreetLearn
Benchmarks: Touchdown
Metrics: Task Completion (TC) Shortest Path Distance (SPD) Success weighted by Edit Distance (SED)
Baselines: RConcat GA ARC-L2S VLN-Transformer ORAR PM-VLN ORAR-BERT
Builds on: VLN-Transformer ORAR BERT Mask-RCNN GPT-2
Closest papers by meaning
- Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.88)
- Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models (NeurIPS 2025 · similarity 0.87)
- Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Method (CVPR 2025 · similarity 0.87)
- GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigation (CVPR 2026 · similarity 0.87)
- FIVE-VLA: Fast and EffectIVE Closed-Loop Autonomous Driving with Recurrent Action Memory (NeurIPS 2026 · similarity 0.86)
- OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.