Research map / Video / Video-language understanding
RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds RoomTour3D, a video-instruction dataset from web room-tour videos with 3D-reconstructed walking trajectories, GPT-4 instructions, and navigable actions, and uses it to boost NaviLLM on VLN tasks.
Problem
Vision-and-language navigation training data is limited in scale and diversity because existing simulators and annotations are manually curated.
Approach
Mines open-ended first-person walking trajectories from web room-tour videos, enriches them with open-vocabulary object tags, depth, room types and COLMAP 3D geometry, and turns them into free-form instructions and geometry-grounded navigable actions for training a generalist VLN agent.
Key results
- NaviLLM+RT3DAction reaches 31.7 SOON Val-U SR and 27.8 Test SR, and 37.4 REVERIE Val-U SR with 36.4 Test SPL, setting state-of-the-art on SOON, R2R and REVERIE.
- In the zero-shot setting on R2R Val Unseen, NaviLLM+RT3D reaches 14.33 SR and 10.86 SPL, outperforming LLaMA2-7B-based LangNav and NavCoT.
- Manual rating of 100 trajectory descriptions gives an average relevance score of 3.08 on a 4-point scale, with 74% rated mostly relevant or perfect.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RoomTour3D description-enriched trajectories | NaviLLM · CVDN Val-U | GP: 6.09 -> 6.96 | Helped |
| Add RoomTour3D action-enriched trajectories | NaviLLM · SOON Val-U | SPL: 28.0 -> 31.7 | Helped |
| Add RoomTour3D action-enriched trajectories | NaviLLM · R2R Val-U | SPL: 56.7 -> 62.4 | Helped |
| Add RoomTour3D action-enriched trajectories | NaviLLM · REVERIE Val-U | SPL: 31.4 -> 37.4 | Helped |
| Add object tags (ablation) | NaviLLM trajectory summarization · REVERIE | SPL: 31.35 -> 34.37 | Helped |
| Add object tags (ablation) | NaviLLM trajectory summarization · SOON | SR: 33.64 -> 32.52 | No clear effect |
| Add depth estimation and bounding-box information (ablation) | NaviLLM trajectory summarization · SOON | SPL: 26.51 -> 30.40 | Helped |
| Add room-type information (ablation) | NaviLLM trajectory summarization · CVDN | GP: 6.49 -> 6.96 | Helped |
| Add RoomTour3D action-instruction data | NaviLLM · REVERIE Test | SPL: 32.3 -> 36.4 | Helped |
Surprising findings
- Adding object variety alone did not help SOON, which the authors attribute to SOON relying only on textual descriptions without bounding boxes.
Limitations
- Description quality is only moderate: average manual relevance of 3.08/4 with 74% mostly or perfectly relevant.
- Room-location annotation accuracy is 85% on 50 manually checked clips.
- Zero-shot performance remains well below commercial GPT-4-based agents such as NavGPT and MapGPT.
Experimental setup
Datasets: RoomTour3D CVDN SOON R2R REVERIE ScanQA LLaVA-23k
Benchmarks: CVDN SOON R2R REVERIE
Metrics: SR SPL GP
Baselines: NaviLLM DUET HAMT BEV-BERT NavGPT MapGPT DiscussNav NavCoT LangNav
Builds on: NaviLLM COLMAP GPT-4 Grounding DINO RAM
Closest papers by meaning
- Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation (NeurIPS 2025 · similarity 0.88)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.88)
- Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions (NeurIPS 2024 · similarity 0.88)
- Scene Map-based Prompt Tuning for Navigation Instruction Generation (CVPR 2025 · similarity 0.87)
- LocoVR: Multiuser Indoor Locomotion Dataset in Virtual Reality (ICLR 2025 · similarity 0.87)
- D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.