Research map / Video / Video-language understanding
Previously on ... From Recaps to Story Summarization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces multimodal story summarization for long TV episodes, using recap shots to auto-generate labels on a new PlotSnap dataset, and a hierarchical Transformer TaleSumm that scores shots and utterances.
Problem
Summarizing the plot of long (about 40-minute) TV episodes into multiple connected sub-stories, where existing summarization methods target short videos or text-only screenplays and lack labels for story-level importance.
Approach
Uses professionally edited TV recaps shown at the start of an episode as weak labels: recap shots are matched to shots in past episodes, and the matches (smoothed temporally) mark important story segments. A two-level hierarchical Transformer with local story groups and group tokens scores shots and dialog utterances across the whole episode.
Key results
- TaleSumm reaches 50.1 video AP and 46.0 dialog AP on the PlotSnap test set, beating A2Summ (33.8 / 31.6) and the video-only and dialog-only baselines.
- On SumMe, TaleSumm reaches F1 57.5 versus 56.2 for PGL-SUM, a state-of-the-art result on that benchmark; on TVSum it is comparable (64.0 F1 vs 63.9).
- Cross-series transfer from 24 to Prison Break gives 36.7 video AP, well above the random baseline of 21.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TaleSumm instead of the strongest video-only baseline | PlotSnap 24 IntraCVT · PlotSnap 24 IntraCVT test | Video AP: 50.6 ± 3.6 -> 50.1 ± 2.8 | No clear effect |
| Use TaleSumm instead of the strongest dialog-only baseline | PlotSnap 24 IntraCVT · PlotSnap 24 IntraCVT test | Dialog AP: 43.4 ± 2.8 -> 46.0 ± 2.1 | Helped |
| Train on 24 and evaluate on Prison Break | TaleSumm · PlotSnap Prison Break X-Series | Video AP: 34.5 -> 36.7 | Helped |
| Train on 24 and evaluate on Prison Break | TaleSumm · PlotSnap Prison Break X-Series | Dialog AP: 38.3 -> 35.7 | Hurt |
| Use TaleSumm on SumMe rather than PGLSUM | Video summarization · SumMe | F1: 56.2 -> 57.5 | Helped |
| Use TaleSumm on TVSum rather than PGLSUM | Video summarization · TVSum | SP: 40.5 -> 26.7 | Hurt |
| Add local story groups and sparse attention instead of an independent-token MLP (ablation) | PlotSnap video and dialog models · PlotSnap 24 architecture ablation | AP: 42.3 -> 53.3; 35.8 -> 47.2 | Helped |
| Use sparse attention with local story groups instead of full attention without groups (ablation) | PlotSnap TaleSumm · PlotSnap 24 architecture ablation | Video AP: 51.8 -> 53.4 | Helped |
| Use sparse attention with local story groups instead of full attention without groups (ablation) | PlotSnap TaleSumm · PlotSnap 24 architecture ablation | Dialog AP: 44.6 -> 47.2 | Helped |
| Add group tokens while using sparse attention (ablation) | PlotSnap TaleSumm · PlotSnap 24 architecture ablation | Video AP: 53.9 ± 3.4 -> 54.2 ± 3.3 | Helped |
| Add group tokens while using sparse attention (ablation) | PlotSnap TaleSumm · PlotSnap 24 architecture ablation | Dialog AP: 48.8 ± 4.6 -> 49.0 ± 4.9 | No clear effect |
| Use sparse attention instead of full attention, retaining group tokens (ablation) | PlotSnap TaleSumm · PlotSnap 24 architecture ablation | Video AP: 51.9 ± 3.7 -> 54.2 ± 3.3 | Helped |
Surprising findings
- Standard video summarization baselines (A2Summ) perform poorly on the long-episode setting, and the full-attention Transformer without local groups underperforms the grouped sparse-attention design.
- TaleSumm is competitive on TVSum but drops in Kendall's and Spearman's correlations relative to PGL-SUM, which the authors attribute to video diversity.
Limitations
- Limited to one genre (crime thriller) and two TV series, so generalization across genres is untested.
- Does not explicitly model characters, faces, or emotions, which are central to stories.
- Early experiments on condensed movies show only limited improvements.
Experimental setup
Datasets: PlotSnap SumMe TVSum Condensed Movies (CMD)
Benchmarks: PlotSnap SumMe TVSum
Metrics: Average Precision F1 Kendall's tau Spearman's rho Cronbach's alpha pairwise F1 Fleiss' kappa
Baselines: PGL-SUM MSVA PreSumm A2Summ Video-Only Dialog-Only
Builds on: A2Summ PreSumm PGL-SUM MSVA SummScreen3D
Closest papers by meaning
- SeriesBench: A Benchmark for Narrative-Driven Drama Series Understanding (CVPR 2025 · similarity 0.85)
- Video Summarization with Large Language Models (CVPR 2025 · similarity 0.84)
- Scaling Up Video Summarization Pretraining with Large Language Models (CVPR 2024 · similarity 0.84)
- V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning (AAAI 2025 · similarity 0.84)
- REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editing (NeurIPS 2025 · similarity 0.82)
- FriendsQA: A New Large-Scale Deep Video Understanding Dataset with Fine-grained Topic Categorization for Story Videos (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.