Research map / Video / Video-language understanding
Learning from Narrated Instruction Videos
Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Šivic, Ivan Laptev, Simon Lacoste-Julien
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Frank-Wolfe relaxation for multiple sequence alignment instead of Poa | step discovery from narrations · Changing tire (MSA objective) | objective value (lower better): 11.30 -> -5.18 | Improved |
| use Frank-Wolfe relaxation instead of Poa | step discovery from narrations · Performing CPR (MSA objective) | objective value (lower better): -3.82 -> -4.51 | Improved |
| use Frank-Wolfe relaxation instead of Poa | step discovery from narrations · Repotting plant (MSA objective) | objective value (lower better): 1.65 -> -3.55 | Improved |
| use Frank-Wolfe relaxation instead of Poa | step discovery from narrations · Making coffee (MSA objective) | objective value (lower better): -2.99 -> -3.86 | Improved |
| use Frank-Wolfe relaxation instead of Poa | step discovery from narrations · Jumping cars (MSA objective) | objective value (lower better): 4.55 -> -4.67 | Improved |
| add text narration constraints to video-only clustering (ablation) | two-stage text+video clustering · Change tire and Perform CPR step localization | F1 score | Improved |
| add text narration constraints to video-only clustering (ablation) | two-stage text+video clustering · Make coffee step localization | F1 score | No clear effect |
| add text narration constraints to video-only clustering (ablation) | two-stage text+video clustering · Jump car step localization | F1 score | Mixed |
| use word2vec similarity instead of WordNet in first stage | two-stage clustering · step localization | F1 score | No clear effect |
| tune regularization lambda away from 1/(N K_pred) | two-stage clustering · five tasks average | F1 score: up to 5% improvement from a good choice of lambda | Mixed |
| use very large caption delays (Delta b, Delta a) > (60, 60) | two-stage clustering · five tasks average | F1 score: drop of more than 5% | Hurt |
| add localization noise to narration constraints (ablation) | two-stage clustering · five tasks average | F1 score | No clear effect |
Closest papers by meaning
- Unsupervised Learning From Narrated Instruction Videos (CVPR 2016 · similarity 0.97)
- StepFormer: Self-Supervised Step Discovery and Localization in Instructional Videos (CVPR 2023 · similarity 0.89)
- Self-Supervised Multi-Task Procedure Learning from Instructional Videos (ECCV 2020 · similarity 0.89)
- SLVMBench: Skill Learning from Video Memory (NeurIPS 2026 · similarity 0.87)
- Detours for Navigating Instructional Videos (CVPR 2024 · similarity 0.86)
- Step Differences in Instructional Video (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
