Research map / Video / Video-language understanding
Video Summarization with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLMVS, which captions video frames with LLaVA-1.5-7B, scores frame importance with Llama-2-13B-chat via in-context learning over local windows, and aggregates global context with self-attention blocks over LLM output embeddings.
Problem
Video summarization relies mainly on visual features and temporal dynamics, which often miss semantics and human-perceived importance, yielding incomplete or incoherent summaries.
Approach
Use a frozen M-LLM to caption frames and a frozen LLM, prompted with in-context examples, to score each frame's importance within a sliding caption window; take the LLM's intermediate query and answer embeddings (not its text output) and train only a self-attention block on top to aggregate global context.
Key results
- LLMVS achieves Kendall's tau 0.253 and Spearman's rho 0.282 on SumMe, versus 0.246/0.274 for the prior best CSTA.
- On TVSum, LLMVS reaches tau 0.211 and rho 0.275, above CSTA's 0.194/0.255.
- Zero-shot transfer trained on SumMe to 50 MR.HiSum videos gives tau 0.440, above PGL-SUM's 0.375.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the global context aggregator to LLM local scoring | LLMVS · SumMe | Kendall's τ: 0.170 -> 0.253 | Helped |
| Add the global context aggregator to LLM local scoring | LLMVS · TVSum | Kendall's τ: 0.051 -> 0.211 | Helped |
| Use captions from M-LLM as input to an LLM rather than direct M-LLM frame scoring | Video summarization · SumMe | Kendall's τ: 0.119 -> 0.170 | Helped |
| Finetune LLaVA for direct frame-importance scoring (ablation) | M-LLM direct scoring · SumMe | Kendall's τ: 0.119 -> 0.140 | Helped |
| Finetune Llama as the local importance scorer (ablation) | Caption-guided LLM scoring · SumMe | Kendall's τ: 0.170 -> 0.181 | Helped |
| Use generic one-sentence captions instead of separate center and background descriptions (ablation) | LLMVS · SumMe | Kendall's τ: 0.241 -> 0.253 | Helped |
| Ask for numerical importance scores instead of textual explanations (ablation) | LLM importance scoring · SumMe | Kendall's τ: 0.239 -> 0.253 | Helped |
| Add query embeddings to answer embeddings (ablation) | LLMVS embedding inputs · SumMe | Kendall's τ: 0.233 -> 0.253 | Helped |
| Add answer embeddings to query embeddings (ablation) | LLMVS embedding inputs · SumMe | Kendall's τ: 0.238 -> 0.253 | Helped |
| Use global self-attention blocks instead of an MLP (ablation) | LLMVS global context aggregator · SumMe | Kendall's τ: 0.182 -> 0.253 | Helped |
| Extract embeddings after RMS Norm instead of after the Linear layer (ablation) | LLMVS · SumMe | Kendall's τ: 0.241 -> 0.253 | Helped |
| Use LLMVS rather than the strongest reported baseline | Video summarization · SumMe | Kendall's τ: 0.246 -> 0.253 | Helped |
Surprising findings
- Zero-shot Llama-2 scoring alone performs poorly on TVSum (tau 0.051), attributed to differing evaluation protocols and subjectivity.
- Extracting embeddings after the RMS Norm layer works better than after the final Linear layer (SumMe tau 0.253 vs 0.241).
Limitations
- Relies on frozen LLM prompting with a fixed set of in-context examples drawn from SumMe.
- Evaluation excludes F1 score, so results are not directly comparable to F1-based prior work.
- TVSum performance gains are modest and the zero-shot LLM baseline is weak on TVSum.
Experimental setup
Datasets: SumMe TVSum
Benchmarks: SumMe TVSum MR.HiSum
Metrics: Kendall's tau Spearman's rho
Baselines: VASNet DSNet-AB DSNet-AF DMASum PGL-SUM MSVA iPTNet CSTA CLIP-It A2Summ SSPVS Argaw et al.
Builds on: LLaVA Llama-2 VASNet CLIP-It A2Summ
Closest papers by meaning
- Scaling Up Video Summarization Pretraining with Large Language Models (CVPR 2024 · similarity 0.93)
- V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning (AAAI 2025 · similarity 0.91)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.88)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.88)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.88)
- Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.