Research map / Video / Video-language understanding
VideoPrism: A Foundational Visual Encoder for Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VideoPrism, a frozen general-purpose video encoder pretrained with video-text contrastive learning followed by masked video modeling with global-local distillation and token shuffling, evaluated across 33 video benchmarks.
Problem
Building a single foundational video encoder that performs well on both appearance-heavy and motion-centric video understanding tasks, using a mix of high-quality and noisy video-text data plus video-only data.
Approach
A two-stage pipeline: contrastive video-text training to obtain semantic video embeddings, then masked video modeling on video-only data that distills both global and token-wise embeddings from the stage-1 teacher, with a token shuffling scheme to prevent copy-and-paste decoding shortcuts.
Key results
- VideoPrism-g with frozen backbone reaches 87.2 top-1 on K400 and 45.5 on MiT, outperforming the best baseline by 4.4 and 5.2 points on VideoGLUE.
- Zero-shot MSRVTT (1K-A) text-to-video R@1 of 52.7 for VideoPrism-g, +8.8 over the previous best.
- Reported state of the art on 31 of 33 video understanding benchmarks; VideoPrism-g achieves VGS 51.25, 13.6% above the second-best model (UMT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale the encoder from VideoPrism-B to VideoPrism-g | VideoPrism · Kinetics-400, MiT, SSv2, Diving48, Charades, ActivityNet, AVA, and AVA-K | benchmark accuracy or mAP: 84.2 -> 87.2 (K400); 40.8 -> 45.5 (MiT); 63.6 -> 68.5 (SSv2); 67.4 -> 71.3 (D48); 40.4 -> 62.3 (Charades); 36.6 -> 37.8 | Helped |
| Add global distillation and token shuffling (ablation) | VideoPrism pretraining · Kinetics-400 | accuracy | Helped |
| Add the proposed data curation and model-design components (ablation) | VideoPrism pretraining · SSv2 | accuracy | Helped |
| Use VideoPrism-B instead of VideoPrism-g | VideoPrism · KABR | macro-accuracy: 63.3 -> 61.6 | Hurt |
Surprising findings
- Video-only masked modeling alone underperforms the stage-1 teacher on appearance-heavy tasks, so global distillation is needed to avoid catastrophic forgetting.
- Captions from VLM/LLM generation (InternVid, Anonymous-Corpus #3) gave the best retrieval performance among the pretraining corpora, despite being noisy.
Limitations
- Pretraining relies on noisy, potentially biased ASR and metadata text.
- Focuses on short clips (16 frames), so long-video understanding is not addressed.
- Frozen-backbone evaluation may not reflect scenarios that benefit from end-to-end finetuning.
Experimental setup
Datasets: WTS-70M YT-Temporal-180M VideoCC InternVid WebLI Anonymous-Corpus #1 Anonymous-Corpus #2 Anonymous-Corpus #3
Benchmarks: Kinetics-400 MiT SSv2 Diving48 Charades ActivityNet AVA AVA-Kinetics MSRVTT VATEX YouCook2 MSRVTT-QA
Metrics: Top-1 accuracy mAP Recall@1 CIDEr WUPS macro-accuracy VideoGLUE score
Baselines: CLIP VATT InternVideo UMT VideoMAE-v2 VideoCoCa CoCa Flamingo FrozenBiLM UMT-L
Builds on: ViViT CoCa BEVT MVD LiT
Closest papers by meaning
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.89)
- $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation (ICML 2025 · similarity 0.89)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.88)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.88)
- Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.