Research map / Video / Video-language understanding
Learning to Ground Instructional Articles in Videos through Narrations
Effrosyni Mavroudi, Triantafyllos Afouras, Lorenzo Torresani
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on wikiHow step pseudo-labels with direct pathway only (no narrations) (ablation) | VINA · HT-Step val | R@1: 30.7 -> 34.0 (+3.3 over TAN*) | Improved |
| add iterative pseudo-label refinement to single-task step training (ablation) | VINA · HT-Step val | R@1 w/o nar.: 34.0 -> 35.8 | Improved |
| add multi-task training with narrations and steps (ablation) | VINA · HTM-Align | R@1: 63.2 -> 64.8 | Improved |
| add multi-task training with narrations and steps, no narrations at inference (ablation) | VINA · HT-Step val | R@1 w/o nar.: 34.0 -> 34.3 | No clear effect |
| add iterative pseudo-labeling to multi-task model (ablation) | VINA · HT-Step val | R@1 w/ nar.: 36.1 -> 39.1 | Improved |
| fuse direct and indirect alignment at inference (ablation) | VINA · HT-Step val | R@1: 34.3 -> 36.1 | Improved |
| replace learned step-narration alignment with off-the-shelf MPNet similarity (ablation) | indirect pathway · HT-Step val | R@1: 30.5 -> 19.0 | Hurt |
| replace learned narration-to-video alignment with ASR timestamps (ablation) | indirect pathway · HT-Step val | R@1: 30.5 -> 27.9 | Hurt |
| use aggressive pseudo-label filtering (gamma 0.65, 91% discarded) instead of gamma 0.40 (ablation) | iterative pseudo-labeling · HT-Step val | R@1 w/ nar.: 36.5 -> 39.1 | Improved |
| use task ids predicted from narrations instead of HowTo100M metadata (ablation) | article pairing · HT-Step val | R@1: 34.3 -> 34.7 | No clear effect |
| add step positional embeddings and separate MLP (ablation) | unimodal encoder · HT-Step val / HTM-Align | R@1: HT-Step w/o nar. 34.0 -> 34.3; HTM-Align 65.9 -> 64.8 | Mixed |
| use VINA instead of TAN | narration alignment · HTM-Align | R@1: 49.4 -> 66.5 | Improved |
Closest papers by meaning
- HT-Step: Aligning Instructional Articles with How-To Videos (NeurIPS 2023 · similarity 0.93)
- Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment (ECCV 2024 · similarity 0.92)
- Learning Procedure-Aware Video Representation From Instructional Videos and Their Narrations (CVPR 2023 · similarity 0.91)
- Detours for Navigating Instructional Videos (CVPR 2024 · similarity 0.88)
- Step Differences in Instructional Video (CVPR 2024 · similarity 0.87)
- SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
