Research map / Video / Video-language understanding
Scaling Up Video Summarization Pretraining with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 250K-pair pseudo-labeled video summarization pretraining dataset (LfVS-P) by prompting LLMs (GPT-3.5) to extract timestamped transcript sentences, and trains an autoregressive Transformer video summarizer with a new 1,200-video human-annotated benchmark (LfVS-T).
Problem
Video summarization is limited by small annotated datasets (TVSum with 50 videos, SumMe with 25), so supervised models overfit and generalize poorly.
Approach
Use LLMs as oracle summarizers over timestamped transcripts of narrated long videos to create large-scale pseudo-ground-truth summary videos, and decode summaries autoregressively as continuous features conditioned on video and text, with random text masking so the model works without narration.
Key results
- On LfVS-T, the proposed model reaches 68.11 F1, versus 66.04 for A2Summ and 62.87 for CLIP-It.
- Fine-tuning on SumMe and TVSum after LfVS-P pretraining gives 60.42 and 72.38 F1, outperforming the prior methods trained from scratch.
- Pseudo-label source ablation on 50K samples: GPT-4 gives 55.96 F1, GPT-3.5-16K 53.44, Llama-2-13B 44.89.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use autoregressive summary decoding instead of classification-based prediction (ablation) | video summarization model · LfVS-T | F1 Score: 63.31 -> 68.11 | Helped |
| increase pretraining data from 10% to 100% of LfVS-P (ablation) | video summarization model · LfVS-T | F1 Score: 53.44 -> 68.11 | Helped |
| use GPT-4 instead of GPT-3.5-16K as the data summarizer (ablation) | video summarization model trained on 50K samples · LfVS-T | F1 Score: 53.44 -> 55.96 | Helped |
| use GPT-3.5-16K instead of Llama-2-13B as the data summarizer (ablation) | video summarization model trained on 50K samples · LfVS-T | F1 Score: 44.89 -> 53.44 | Helped |
| add text input during pretraining (ablation) | video summarization model · LfVS-T | F1 Score: 66.59 -> 68.11 | Helped |
| add the video encoder (ablation) | video summarization model · LfVS-T | F1 Score: 63.54 -> 68.11 | Helped |
| add the text encoder (ablation) | video summarization model · LfVS-T | F1 Score: 67.49 -> 68.11 | Helped |
| use cross-modal attention instead of concatenating video and text features (ablation) | video summarization model · LfVS-T | F1 Score: 67.72 -> 68.11 | Helped |
| pretrain on LfVS-P and fine-tune on the benchmark training split (ablation) | Ours · SumMe | F1 Score: 56.72 -> 60.42 | Helped |
| pretrain on LfVS-P and fine-tune on the benchmark training split (ablation) | Ours · TVSum | F1 Score: 65.76 -> 72.38 | Helped |
Surprising findings
- Llama-2-13B struggled to follow prompt instructions and produced low-quality pseudo-summaries, performing far below GPT-based summarizers.
Limitations
- Some baselines (CLIP-It, iPTNet) were reimplemented because official code was not public, so comparisons depend on reimplementation.
- Pseudo-labels are noisy from automatic transcription and LLM summarization, and small-scale training is sensitive to this noise.
- Relies on narrated videos with speech-visual alignment, which restricts the source data.
Experimental setup
Datasets: HowTo100M LfVS-P TVSum SumMe
Benchmarks: LfVS-T TVSum SumMe
Metrics: F1 score Kendall's tau Spearman's rho
Baselines: CLIP-It TL:DW? iPTNet A2Summ Human
Builds on: CLIP-It TL:DW? A2Summ HowTo100M Whisper
Closest papers by meaning
- Video Summarization with Large Language Models (CVPR 2025 · similarity 0.93)
- V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning (AAAI 2025 · similarity 0.89)
- World Model on Million-Length Video And Language With Blockwise RingAttention (ICLR 2025 · similarity 0.88)
- Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs (CVPR 2025 · similarity 0.86)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.86)
- MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.