Research map / Video / Video segmentation and tracking
Video Summarization Using Denoising Diffusion Probabilistic Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a DDPM-based generative video summarization method that denoises importance scores conditioned on frame features, warm-started from an unsupervised summarizer (DR-DSN) to handle limited data and annotator subjectivity.
Problem
Supervised video summarization with discriminative frame-importance predictors is sensitive to inconsistent subjective annotations across annotators and overfits limited training data, generalizing poorly across domains.
Approach
Model frame-level importance scores with a DDPM conditioned on video frame features, trained directly on raw per-annotator scores; reduce the maximum noise steps (t=200) and start denoising from an unsupervised DR-DSN output instead of pure Gaussian noise.
Key results
- Achieves 64.8 (TVSum canonical) and 58.7 (SumMe canonical) F-score, best among compared methods in all settings.
- Reaches 46.1 F-score on FPVSum versus 35.3 for FPVS.
- Ablation: removing the unsupervised warm start (w/o unsup) drops TVSum canonical to 64.1 and FPVSum to 42.3; removing DDPM (w/o DDPM) drops TVSum canonical to 62.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DDPM to the unsupervised summarizer (ablation) | Video Summarization Using DDPM · TVSum canonical | F-score: 62.0 -> 64.8 | Helped |
| Add DDPM to the unsupervised summarizer (ablation) | Video Summarization Using DDPM · TVSum augmented | F-score: 61.5 -> 65.0 | Helped |
| Add DDPM to the unsupervised summarizer (ablation) | Video Summarization Using DDPM · TVSum transfer | F-score: 55.3 -> 60.9 | Helped |
| Add DDPM to the unsupervised summarizer (ablation) | Video Summarization Using DDPM · SumMe canonical | F-score: 51.1 -> 58.7 | Helped |
| Add DDPM to the unsupervised summarizer (ablation) | Video Summarization Using DDPM · SumMe augmented | F-score: 51.1 -> 59.2 | Helped |
| Add DDPM to the unsupervised summarizer (ablation) | Video Summarization Using DDPM · SumMe transfer | F-score: 46.5 -> 50.3 | Helped |
| Add DDPM to the unsupervised summarizer (ablation) | Video Summarization Using DDPM · FPVSum | F-score: 39.1 -> 46.1 | Helped |
| Add the unsupervised summarizer as the starting point for denoising (ablation) | Video Summarization Using DDPM · TVSum canonical | F-score: 51.1 -> 58.7 | Helped |
| Add the unsupervised summarizer as the starting point for denoising (ablation) | Video Summarization Using DDPM · TVSum augmented | F-score: 57.5 -> 65.0 | Helped |
| Add the unsupervised summarizer as the starting point for denoising (ablation) | Video Summarization Using DDPM · TVSum transfer | F-score: 57.6 -> 60.9 | Helped |
| Add the unsupervised summarizer as the starting point for denoising (ablation) | Video Summarization Using DDPM · SumMe canonical | F-score: 51.9 -> 58.7 | Helped |
| Add the unsupervised summarizer as the starting point for denoising (ablation) | Video Summarization Using DDPM · SumMe augmented | F-score: 57.5 -> 59.2 | Helped |
Surprising findings
- Denoising from pure Gaussian noise (t=1000) without the unsupervised warm start degrades performance, indicating DDPM alone struggles with limited data.
- The unsupervised DR-DSN was chosen as the warm start; other unsupervised models (SUM-GAN-AAE, AC-SUM-GAN) gave similar results in some settings.
Limitations
- Performance degrades sharply when the number of denoising steps is 50 or 1000, showing sensitivity to the noise-step hyperparameter.
- Relies on an unsupervised model to warm-start denoising; DDPM alone underperforms under limited data.
- Evaluated only on three small benchmarks with no hardware or compute details reported.
Experimental setup
Datasets: TVSum SumMe FPVSum YouTube OVP
Benchmarks: TVSum SumMe FPVSum
Metrics: F-score Kendall's tau Spearman's rho
Baselines: dppLSTM VASNet SUM-FCN DSNet RSGN RR-STG CFT-GIB SSPVS MPFN DR-DSN SUM-GAN-AAE FPVS
Builds on: DDPM DR-DSN UNet
Closest papers by meaning
- Video Summarization with Large Language Models (CVPR 2025 · similarity 0.86)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.85)
- Scaling Up Video Summarization Pretraining with Large Language Models (CVPR 2024 · similarity 0.85)
- V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning (AAAI 2025 · similarity 0.85)
- FIFO-Diffusion: Generating Infinite Videos from Text without Training (NeurIPS 2024 · similarity 0.85)
- Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.