Research map / Video / Video-language understanding
V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Instruct-V2Xum, a 30k-video cross-modal summarization dataset, and V2Xum-LLaMA, which unifies V2V, V2T and V2VT summarization in one LLaMA-2/Vicuna decoder using interleaved temporal prompts, plus CLIP-based F metrics.
Problem
Video summarization suffers from few source videos for training large VLMs, most datasets are video-to-video only, and existing text summaries (e.g., from ActivityNetCap dense captions) are redundant and not true summaries.
Approach
Interleave CLIP visual tokens with zero-padded natural-language temporal prompts and generate video summaries as frame-index text inside a single LLM decoder, controlled by task instructions, with no task-specific regression heads; also proposes F_CLIP and Cross-F_CLIP metrics that reward semantically similar frame matches.
Key results
- On VideoXum, V2Xum-LLaMA-7B reaches V2V F1 29.0 and V2T CIDEr 26.9, versus 23.5 F1 for VTSUM-BLIP; 13B reaches F1 31.6.
- On TVSum and SumMe, V2Xum-LLaMA achieves Spearman 0.293 on TVSum and 0.378 on SumMe, above the best prior VSUM-BLIP (0.261 and 0.365).
- The ablation shows removing the pretrained vision adapter drops V2V F1 from 29.0 to 3.7 and CIDEr from 26.9 to 9.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use V2Xum-LLaMA-7B instead of VTSUM-BLIP | VideoXum V2V summarization · VideoXum | F1: 23.5 -> 29.0 | Helped |
| increase the decoder from 7B to 13B | V2Xum-LLaMA · VideoXum V2V | F1: 29.0 -> 31.6 | Helped |
| increase the decoder from 7B to 13B | V2Xum-LLaMA · VideoXum V2T | BLEU-4: 5.8 -> 5.7 | No clear effect |
| increase the decoder from 7B to 13B | V2Xum-LLaMA · Instruct-V2Xum test V2T | ROUGE-L: 26.9 -> 27.0 | Helped |
| add simultaneous video-text summarization training (ablation) | V2Xum-LLaMA · VideoXum | F1-Score: 25.1 -> 29.0 | Helped |
| add Instruct-V2Xum training data (ablation) | V2Xum-LLaMA · VideoXum | BLEU-4: 4.9 -> 5.8 | Helped |
| fully fine-tune parameters instead of using parameter-efficient fine-tuning (ablation) | V2Xum-LLaMA · VideoXum | F1-Score: 23.4 -> 29.0 | Helped |
| add temporal prompts (ablation) | V2Xum-LLaMA · VideoXum | F1-Score: 23.9 -> 29.0 | Helped |
| pretrain the vision adapter (ablation) | V2Xum-LLaMA · VideoXum | F1-Score: 3.7 -> 29.0 | Helped |
| add Instruct-V2Xum training data (ablation) | V2Xum-LLaMA · VideoXum | Cross-F_CLIP: 0.252 -> 0.253 | No clear effect |
Surprising findings
- Removing the pretrained vision adapter collapses V2V F1 to 3.7, indicating adapter pre-training is critical.
- V2Xum-LLaMA-13B underperforms 7B on V2T BLEU/CIDEr in VideoXum, which the authors attribute to limited training data (8k examples) for large decoders.
Limitations
- Limited training data (8k in VideoXum) is insufficient for training 13B models on V2T summarization.
- Text summaries and video summaries in Instruct-V2Xum are synthesized by GPT-4/LLaVA with only partial human verification.
- Evaluation metrics F_CLIP and Cross-F_CLIP rely on CLIP similarity, and subjectivity of summary quality remains.
Experimental setup
Datasets: Instruct-V2Xum VideoXum TVSum SumMe InternVid LCS-558K
Benchmarks: VideoXum Instruct-V2Xum TVSum SumMe
Metrics: BLEU-4 METEOR ROUGE-L CIDEr F1 Spearman Kendall F_CLIP Cross-F_CLIP
Baselines: Frozen-BLIP VSUM-BLIP TSUM-BLIP VTSUM-BLIP Vid2Seq DENSE DVC-D-A Bi-LSTM+TempoAttn Masked Transformer Support-Set dppLSTM DSN
Builds on: LLaMA-2 Vicuna CLIP Vid2Seq BLIP-2
Closest papers by meaning
- Video Summarization with Large Language Models (CVPR 2025 · similarity 0.91)
- Scaling Up Video Summarization Pretraining with Large Language Models (CVPR 2024 · similarity 0.89)
- MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos (CVPR 2024 · similarity 0.88)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.88)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.87)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.