Research map / Video / Video-language understanding
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Panda-70M, a 70M-clip video-caption dataset by splitting HD-VILA-100M videos semantically, captioning clips with multiple cross-modality VLM teachers, and selecting captions with a fine-tuned UMT retriever; also distills a two-branch student captioner.
Problem
High-quality video-text data is hard to collect: manual annotation is slow and ASR subtitles often miss the main content and action of videos.
Approach
Caption each semantically consistent clip with several teacher models that take different modality inputs (frames, title, description, subtitles, video), then pick the best caption with a retriever fine-tuned on human-selected captions, and distill the ensemble into a single student model.
Key results
- Video-LLaMA pretrained on Panda-2M improves B-4 on MSR-VTT from 5.8 to 23.5 zero-shot captioning, with the student trained on Panda-70M reaching 25.4.
- UMT pretrained on Panda-5M improves zero-shot text-to-video R@1 on MSR-VTT from 30.2 to 37.2.
- The distilled student (video+text inputs) reaches 21.4% preference ratio, above any single teacher (best single teacher 10.7%), versus 23.3% for all teachers combined.
- AnimateDiff trained on Panda-2M lowers UCF101 FVD from 499.3 to 421.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain Video-LLaMA on Panda-2M instead of the official pretraining data | Video-LLaMA · MSR-VTT | BLEU-4: 5.8 -> 23.5 | Helped |
| pretrain Video-LLaMA on Panda-2M instead of the official pretraining data | Video-LLaMA · MSVD | BLEU-4: 12.7 -> 31.2 | Helped |
| train AnimateDiff with Panda-2M rather than its official data | AnimateDiff · UCF101 | FVD: 499.3 -> 421.9 | Helped |
| train AnimateDiff with Panda-2M rather than its official data | AnimateDiff · MSR-VTT | CLIPSim: 0.2869 -> 0.2880 | Helped |
| add text inputs to the student captioning model alongside video input (ablation) | Student captioning model · caption preference user study | preference ratio: +3.0% preference ratio | Helped |
Surprising findings
- No single teacher model was best for more than 31% of videos in the human study, while 84.7% of videos had at least one good caption across teachers.
- Using an LLM to summarize teacher captions was found to propagate errors from noisy VLM outputs, so the paper avoided it.
Limitations
- Videos come mostly from HD-VILA-100M and are vocal-intensive (news, TV, documentary, instructional), limiting unvocal content.
- Semantic-consistency filtering yields short clips (about 8.5s average), limiting long-video and dense captioning use.
- Human agreement on best caption is only 44.9% R@1, indicating subjective labels.
Experimental setup
Datasets: HD-VILA-100M Panda-70M MSR-VTT MSVD DiDeMo UCF101 CC3M WebVid
Benchmarks: MSR-VTT MSVD DiDeMo UCF101
Metrics: BLEU-4 ROUGE-L METEOR CIDEr BERTScore R@1 R@5 R@10 FVD CLIPSim
Baselines: Video-LLaMA (official weights) VideoChat VideoChat Text BLIP-2 MiniGPT-4 UMT (official weights) AnimateDiff (official weights) CogVideo ModelScope VideoLDM
Builds on: Unmasked Teacher Video-LLaMA BLIP-2 MiniGPT-4 AnimateDiff
Closest papers by meaning
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.90)
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.89)
- Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.89)
- AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025 · similarity 0.89)
- LVD-2M: A Long-take Video Dataset with Temporally Dense Captions (NeurIPS 2024 · similarity 0.88)
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.