Research map / Reinforcement learning / Visual and representation RL
Semantic Visual Navigation by Watching YouTube Videos
Matthew Chang, Arjun Gupta, Saurabh Gupta
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn value function from YouTube videos with Q-learning on pseudo-labeled actions instead of end-to-end RL with dense reward and far more interaction | hierarchical ObjectGoal policy · Gibson tiny val Oracle Stop | SPL: 0.29 -> 0.53 | Improved |
| learn value via Q-learning from videos instead of behavior cloning on same pseudo-labeled videos | navigation policy · Gibson tiny val Oracle Stop | SPL: 0.25 -> 0.53 | Improved |
| use real YouTube videos instead of synthetic videos for VLV | VLV · Gibson tiny val Oracle Stop | SPL: 0.48 -> 0.53 | Improved |
| add VLV Q-learning objective jointly to strong map-supervised value training | Strong Supervision Values · Gibson tiny val Oracle Stop | SPL: 0.55 -> 0.57 | Improved |
| use true actions rather than inverse-model pseudo-labels (ablation) | VLV base · Gibson train envs Oracle Stop | SPL: 0.40 -> 0.41 | No clear effect |
| use ground-truth detections rather than Mask RCNN (ablation) | VLV base · Gibson train envs Oracle Stop | SPL: 0.40 -> 0.40 | No clear effect |
| use optimal trajectories rather than noisy ones (ablation) | VLV base · Gibson train envs Oracle Stop | SPL: 0.40 -> 0.43 | No clear effect |
| add Mask RCNN score when sampling short-term goals (ablation) | VLV base · Gibson train envs Oracle Stop | SPL: 0.40 -> 0.46 | Improved |
| use 360 degree images at training time (ablation) | VLV base · Gibson train envs Oracle Stop | SPL: 0.40 -> 0.47 | Improved |
| Q-learning with pseudo-labeled actions instead of policy evaluation without actions (ablation) | value learning · branching environment | reaches near goal | Improved |
| fine-tune behavior cloning policy with dense-reward RL | Behavior Cloning · Gibson tiny val Oracle Stop | SPL: 0.36 -> 0.24 | Hurt |
Closest papers by meaning
- Learning Vision-and-Language Navigation from YouTube Videos (ICCV 2023 · similarity 0.89)
- Bridging the Visual Semantic Gap in VLN via Semantically Richer Instructions (ECCV 2022 · similarity 0.88)
- LongVideo-R1: Smart Navigation for Low-cost Long Video Understanding (CVPR 2026 · similarity 0.87)
- Grounding Video Models to Actions through Goal Conditioned Exploration (ICLR 2025 · similarity 0.87)
- Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation (NeurIPS 2025 · similarity 0.86)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
