Research map / Video / Video-language understanding
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ShareGPT4Video (40K GPT4V dense video captions via a differential sliding-window captioning strategy), ShareCaptioner-Video (a captioner fine-tuned from IXC2-4KHD used to annotate 4.8M videos), and ShareGPT4Video-8B, an LVLM that reports SOTA on TempCompass, VideoBench, and MVBench.
Problem
Existing video data has brief captions, and naive multi-frame or frame-concatenation prompting of GPT4V yields temporally confused, less detailed captions, limiting video understanding by LVLMs and controllable text-to-video generation.
Approach
A differential sliding-window captioning pipeline: GPT4V describes changes between consecutive keyframes given the previous frame, its differential caption, and the current frame; GPT-4 then summarizes all differential captions into one timestamped video caption. The same design lets a captioner be trained to caption arbitrary videos and re-caption sub-clips by reusing differential captions.
Key results
- ShareGPT4Video-8B reaches 61.5% average on TempCompass, an 11.6-point increase over VideoLLaVA-7B (49.9).
- ShareGPT4Video-8B averages 41.2 on VideoBench and 51.2 on MVBench, surpassing prior best by 2.7 and 8.2 points on average.
- In human preference voting, ShareCaptioner-Video is roughly on par with GPT4V (36.5% vs 37.8% preferred, 23.7% comparable).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace an equivalent amount of VideoChatGPT-100K captions with 28K ShareGPT4Video captions | VideoLLaVA-7B · VideoBench, MVBench, and TempCompass | average performance: +1.1 | Helped |
| Replace an equivalent amount of VideoChatGPT-100K captions with 28K ShareGPT4Video captions | LLaMA-VID-7B · VideoBench, MVBench, and TempCompass | average performance: +2.0 | Helped |
| Replace an equivalent amount of VideoChatGPT-100K captions with 28K ShareGPT4Video captions | LLaMA-VID-13B · VideoBench, MVBench, and TempCompass | average performance: +2.3 | Helped |
| Train with detailed ShareGPT4Video captions rather than short captions | Text-to-video model · 10-second text-to-video generation | complex prompt adherence | Helped |
| Add short captions to VQA training data (ablation) | ShareGPT4Video-8B · Video understanding benchmarks | performance | Mixed |
| Add detailed ShareGPT4Video captions to VQA training data (ablation) | ShareGPT4Video-8B · Video understanding benchmarks, especially temporal comprehension | performance | Helped |
| Unlock the vision encoder during training with detailed captions (ablation) | ShareGPT4Video-8B · Video understanding benchmarks | modality alignment | Helped |
Surprising findings
- Adding short captions on top of VQA data may not yield gains and can even degrade some benchmarks.
- Naive multi-frame and frame-concatenation GPT4V captioning is unstable and degrades as frame count grows.
Limitations
- Cannot incorporate audio information since GPT4V lacks audio input.
- Captions are not manually verified for socially biased content, and videos may contain human faces.
Experimental setup
Datasets: Panda-70M Ego4D BDD100K VideoChatGPT-100K CLEVRER EGO-QA NextQA TGIF-Transition MixKit Pexels Pixabay
Benchmarks: TempCompass VideoBench MVBench
Metrics: accuracy
Baselines: VideoLLaVA-7B LLaMA-VID-7B LLaMA-VID-13B VideoChatGPT-7B GPT4V Panda-Student
Builds on: LLaVA-1.5 IG-VLM Open-Sora-Plan Latte LLaMA-VID
Closest papers by meaning
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.89)
- LVD-2M: A Long-take Video Dataset with Temporally Dense Captions (NeurIPS 2024 · similarity 0.87)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.86)
- Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.86)
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.86)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.