Research map / Video / Video-language understanding
Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Shot2Story, a multi-shot video benchmark with shot-level visual and narration captions, GPT-4-assisted and human-verified summaries, and a QA set, and benchmarks MiniGPT-4 and VideoChat2 based models with ASR input.
Problem
Existing video captioning and QA benchmarks annotate coarse holistic captions and ignore shot structure and human speech, so they do not test fine-grained multi-shot, audio-visual story understanding.
Approach
Annotate shot-level visual captions and narration (speech) captions for each shot of short multi-shot web videos, build LLM-drafted then human-verified video summaries from those captions, and derive a multi-shot QA benchmark; train shot-aware summarizers that feed per-shot frame tokens plus ASR text into an LLM.
Key results
- VideoChat2-SUM-shot reaches 40.5 overall accuracy on Shot2Story-QA versus 14.5 for VideoChat2 with captions+QA instruction tuning.
- VideoChat2-SUM-shot scores 58.5 on MSRVTT-QA and 47.1 on ActivityNet-QA in zero-shot setting, beating 5 of 6 and 4 of 6 video-VLMs respectively.
- Adding ASR text improves VideoChat2-C single-shot captioning CIDEr from 23.7 to 43.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ASR text to VideoChat2-C (ablation) | VideoChat2-C · Shot2Story shot captioning | CIDEr: 23.7 -> 43.9 | Helped |
| add ASR text to MiniGPT4-SUM-shot (ablation) | MiniGPT4-SUM-shot · Shot2Story video summarization | CIDEr: 4.8 -> 7.6 | Helped |
| add audio modality while keeping ASR text (ablation) | VAST · Shot2Story shot captioning | CIDEr: 33.8 -> 34.0 | No clear effect |
| use explicit shot structure instead of holistic frame sampling (ablation) | MiniGPT4 video summarization · Shot2Story video summarization | CIDEr: 2.8 -> 7.6 | Helped |
| use VideoChat2 backbone instead of MiniGPT4 (ablation) | SUM-shot video summarization · Shot2Story video summarization | CIDEr: 7.6 -> 9.0 | Helped |
| use VideoChat2-SUM-shot summaries instead of direct video input | zero-shot QA system · MSRVTT-QA | accuracy: VideoChat2-SUM-shot: 58.5; Video-LLaVA: 59.2 | Mixed |
| use VideoChat2-SUM-shot summaries instead of direct video input | zero-shot QA system · ActivityNet-QA | accuracy: 47.1 vs 49.1 | Hurt |
| use VideoChat2 rather than MiniGPT4 as the SUM-shot backbone (ablation) | summary-based Shot2Story-QA · Shot2Story-QA | overall accuracy: 32.5 -> 40.5 | Helped |
Surprising findings
- Summaries from a model trained only on Shot2Story transfer competitively to out-of-domain MSRVTT-QA and longer ActivityNet-QA, despite no instruction tuning on QA data.
- Video-ChatGPT obtains very low CIDEr (about 1.0) on summarization, far below shot-structured models.
Limitations
- Summaries still contain hallucinations and omit small details; annotations skew toward human-centric activities.
- Training requires 8x2 A100-80G GPUs, which the authors acknowledge as a barrier.
- Over 80% of single-shot visual captions and over 40% of summaries needed manual correction, so annotation is costly.
Experimental setup
Datasets: Shot2Story HD-VILA-100M MSRVTT-QA ActivityNet-QA
Benchmarks: Shot2Story MSRVTT-QA ActivityNet-QA Shot2Story-QA
Metrics: BLEU@4 METEOR ROUGE CIDEr accuracy GPT-3.5 judged accuracy
Baselines: VAST MiniGPT4-C VideoChat2-C Video-ChatGPT LLaMA-VID Video-LLaVA VideoChat MovieChat
Builds on: MiniGPT-4 VideoChat2 VAST Video-ChatGPT
Closest papers by meaning
- VideoPrism: A Foundational Visual Encoder for Video Understanding (ICML 2024 · similarity 0.88)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.87)
- Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.87)
- AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025 · similarity 0.87)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.86)
- Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.