Research map / Video / Video-language understanding
TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency
Medhini Narasimhan, Arsha Nagrani, Chen Sun, Michael Rubinstein, Trevor Darrell, Anna Rohrbach, Cordelia Schmid
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-modal saliency to task relevance for pseudo summaries (ablation) | IV-Sum · WikiHow Summaries val | F-Score: 64.1 -> 67.9 | Improved |
| add task relevance to cross-modal saliency for pseudo summaries (ablation) | IV-Sum · WikiHow Summaries val | F-Score: 61.0 -> 67.9 | Improved |
| pseudo summaries instead of step-localization annotations (ablation) | IV-Sum · WikiHow Summaries val | F-Score: 57.6 -> 66.8 | Improved |
| add ASR transcript input (ablation) | IV-Sum · WikiHow Summaries test | F-Score: 65.2 -> 67.3 | Improved |
| fine-tune S3D mixed_5 layers instead of freezing (ablation) | IV-Sum · WikiHow Summaries val | F-Score: 65.8 -> 67.9 | Improved |
| 24-layer transformer instead of 12 (ablation) | IV-Sum · WikiHow Summaries val | F-Score: 66.7 -> 67.9 | Improved |
| transformer instead of MLP scorer (ablation) | IV-Sum · WikiHow Summaries val | F-Score: 32.1 -> 67.9 | Improved |
| add diversity loss (ablation) | IV-Sum · WikiHow Summaries val | F-Score: 67.9 -> 61.2 | Hurt |
| add reconstruction loss (ablation) | IV-Sum · WikiHow Summaries val | F-Score: 67.9 -> 67.6 | No clear effect |
| MIL-NCE S3D segment features instead of CLIP frame features (ablation) | IV-Sum · WikiHow Summaries test | F-Score: 62.0 -> 67.3 | Improved |
| train CLIP-It on pseudo summaries instead of generic datasets | CLIP-It with ASR · WikiHow Summaries test | F-Score: 27.2 -> 61.8 | Improved |
| IV-Sum trained on pseudo summaries vs pseudo summary generation itself | IV-Sum · WikiHow Summaries subset (15 tasks) | F-Score: 38.0 -> 42.0 | Improved |
Closest papers by meaning
- Query-Focused Video Summarization: Dataset, Evaluation, and a Memory Network Based Approach (CVPR 2017 · similarity 0.89)
- V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning (AAAI 2025 · similarity 0.88)
- Video Summarization via Semantic Attended Networks (AAAI 2018 · similarity 0.88)
- Video Summarization by Learning From Unpaired Data (CVPR 2019 · similarity 0.88)
- Video Summarization with Large Language Models (CVPR 2025 · similarity 0.87)
- Detours for Navigating Instructional Videos (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
