Research map / Video / Video-language understanding
Detours for Navigating Instructional Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the video detours task for navigating instructional cooking videos and VidDetours, an LLaMA2-13B-based model that retrieves detour videos and localizes detour windows conditioned on a source video and a query, trained on a narration-derived weakly supervised dataset.
Problem
Finding a related how-to video and temporal segment that lets a user alter their current recipe step (ingredients, tools, techniques) given the source video watched so far and a natural language query.
Approach
Encode source video tokens, the query, and candidate video tokens jointly in an LLM sequence encoder for early fusion of viewing context and query, with retrieval and localization heads; training data is mined from HowTo100M narrations using LLM-generated summaries and detour annotations.
Key results
- Detour video retrieval: VidDetours achieves R@5 17.6 and R@50 62.4 with median rank 30, versus 9.7 R@5 for the best InternVideo baseline.
- Detour window localization: VidDetours reaches mean R@1 12.8 vs 9.6 for STALE and 9.4 for VSLNet/UMT.
- Manual inspection estimates the weakly-supervised training data is 85% satisfactory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VidDetours rather than the strongest baseline for detour video retrieval | Detour video retrieval · Detour video retrieval | R@5: 9.7 -> 17.6 | Helped |
| Use VidDetours rather than the strongest baseline for detour video retrieval | Detour video retrieval · Detour video retrieval | R@50: 27.2 -> 62.4 | Helped |
| Use VidDetours rather than the strongest baseline for detour window localization | Detour window localization · Detour window localization | Mean R@1: 10.2 -> 12.8 | Helped |
| Use VidDetours rather than the strongest baseline for detour window localization | Detour window localization · Detour window localization | R@1, IoU=0.3: 12.7 -> 16.7 | Helped |
| Add hard-negative sampling (ablation) | VidDetours · Detour video retrieval | R@5: 16.5 -> 17.6 | Helped |
| Add hard-negative sampling (ablation) | VidDetours · Detour video retrieval | MedR: 55 -> 30 | Helped |
| Use LLM-generated timestamps instead of parser-assigned timestamps (ablation) | VidDetours training data · Detour video retrieval | R@5: 13.9 -> 17.6 | Helped |
| Use LLM-generated timestamps instead of parser-assigned timestamps (ablation) | VidDetours training data · Detour window localization | Mean R@1: 11.6 -> 12.8 | Helped |
Surprising findings
- Text-only baselines are weak, but multimodal LLM dense captions (Vid2Seq) and narration-based supervision outperform keystep and caption-based supervision.
Limitations
- Training data is noisy due to LLM errors and misaligned narrations (about 85% satisfactory by manual inspection).
- Localization performance remains low at high IoU thresholds (R@1 at IoU 0.7 is 2.8).
- Evaluated only on the cooking domain.
Experimental setup
Datasets: HowTo100M Detours weakly-supervised training set (D^tr_D)
Benchmarks: Detours test set (16,207 manually annotated instances)
Metrics: recall@5 recall@10 recall@50 median rank recall@1 at IoU 0.3/0.5/0.7 mean recall@1
Baselines: CLIP CLIP-Hitchhiker InternVideo Distant Supervision Multi-modal LLM (Vid2Seq) CoVR 2D-TAN VSLNet UMT STALE
Builds on: VSLNet InternVideo HowTo100M LLaMA 2 CoVR
Closest papers by meaning
- $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation (ICML 2025 · similarity 0.86)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.86)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.86)
- SLVMBench: Skill Learning from Video Memory (NeurIPS 2026 · similarity 0.86)
- Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision (ICLR 2025 · similarity 0.86)
- Step Differences in Instructional Video (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.