Research map / Video / Video-language understanding
MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMSum, a 5,100-video YouTube dataset with human-verified video and text summaries across 170 subcategories, plus benchmarks for video, text, and multimodal summarization and a thumbnail generation application.
Problem
Existing multimodal summarization with multimodal output (MSMO) datasets are small, poorly maintained (many links dead), uncategorized, and lack human verification, hindering research on summarizing videos into both keyframes and text.
Approach
A curated, human-verified, categorized YouTube dataset with author-provided segment-level visual and textual summaries, plus a baseline pipeline (hierarchical BERT segmentation with a multimodal summarizer) and a thumbnail generation application built on MSMO outputs.
Key results
- On MMSum whole-video setting, the proposed MSMO pipeline reaches PSNR 25.28 and SSIM 0.697 for keyframe selection vs. 9.23 PSNR for VSUMM.
- Text summarization: the proposed model achieves BLEU-1 32.61 and BertScore 74.46 in whole-video setting, above the best baseline (BART-large-XSUM, 30.91 BLEU-1; 69.56 BertScore).
- MSMO combined pipeline (LGSS+VSUMM+T5 baseline) is outperformed by the proposed method, e.g., PSNR 36.370 vs 16.234 and BertScore 85.19 vs 62.77.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed video summarization method instead of VSUMM | MMSum video summarization, whole-video · MMSum, whole-video | RMSE: 0.279 -> 0.112 | Helped |
| Use the proposed video summarization method instead of VSUMM | MMSum video summarization, segment-level · MMSum, segment-level | RMSE: 0.122 -> 0.091 | Helped |
| Use the proposed text summarization method instead of BART-large-XSUM | MMSum text summarization, whole-video · MMSum, whole-video | BLEU-1: 30.91 -> 32.61 | Helped |
| Use the proposed text summarization method instead of BART-large-XSUM | MMSum text summarization, segment-level · MMSum, segment-level | ROUGE-L: 7.13 -> 13.24 | Helped |
| Use the proposed MSMO method instead of LGSS + VSUMM + T5 | MMSum MSMO · MMSum | BLEU: 27.35 -> 33.36 | Helped |
| Use the proposed MSMO method instead of LGSS + VSUMM + BART-large-CNN | MMSum MSMO · MMSum | BertScore: 64.44 -> 85.19 | Helped |
| Use the proposed MSMO method instead of LGSS + VSUMM + T5 | MMSum MSMO · MMSum | PSNR: 16.234 -> 36.370 | Helped |
Surprising findings
- ROUGE scores fail to capture performance differences that other metrics reveal, since ROUGE ignores semantics and factual accuracy.
Limitations
- No publicly available MSMO baselines existed, so baselines were reimplemented by the authors.
- Thumbnail generation evaluation lacks quantitative metrics; relies on random font and placement choices.
- Many existing MSMO datasets (e.g., VMSMO) had invalid video links, limiting comparison.
Experimental setup
Datasets: MMSum
Benchmarks: MMSum
Metrics: RMSE PSNR SSIM SRE precision recall F1 BLEU ROUGE-1 ROUGE-2
Baselines: Uniform Sampling K-means Clustering VSUMM BERT2BERT BART-large-CNN BART-large-XSUM T5 Pegasus
Builds on: VSUMM BART T5 hierarchical BERT
Closest papers by meaning
- V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning (AAAI 2025 · similarity 0.88)
- Video Summarization with Large Language Models (CVPR 2025 · similarity 0.87)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.86)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.86)
- Scaling Up Video Summarization Pretraining with Large Language Models (CVPR 2024 · similarity 0.86)
- DIUSum: Dynamic Image Utilization for Multimodal Summarization (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.