Research map / Video / Video-language understanding
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InternVid, a 234M-clip video-text dataset with LLM-assisted multiscale captions, and ViCLIP, a ViT-L video-text model trained on it that achieves strong zero-shot action recognition and retrieval.
Problem
Video-language learning lacks large-scale video-text datasets with high video-text correspondence; existing ASR-based or alt-text datasets are either noisy or small and low in action content.
Approach
Build a large web-scale video-text dataset by generating clip captions with a multiscale pipeline (BLIP2 on the middle frame for coarse captions; Tag2Text frame captions summarized by an LLM for fine captions), and train a CLIP-initialized video encoder (ViCLIP) with masked contrastive learning on it.
Key results
- ViCLIP trained on InternVid-10M-FLT reaches 75.7 (K400), 73.5 (K600), 66.4 (K700) zero-shot average accuracy, exceeding EVA-CLIP-E and WebVid-trained ViCLIP.
- Text-to-video baseline with WebVid10M plus InternVid-Aesthetics-18M reduces FVD from 705.3 to 616.5 on UCF-101 relative to WebVid10M alone.
- Fine-tuned K400 accuracy of ViCLIP with InternVid-200M+K710 is 88.7 and SthSthV2 is 74.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train ViCLIP on InternVid-10M-FLT rather than WebVid10M | ViCLIP · Kinetics-400/600/700 zero-shot action recognition | AVG accuracy: 71.03 -> 75.70; 69.84 -> 73.53; 61.86 -> 66.38 | Helped |
| Increase pretraining data from InternVid-10M-FLT to InternVid-200M (ablation) | ViCLIP · Kinetics zero-shot action recognition | AVG accuracy: 75.70 -> 71.09; 73.53 -> 69.34; 66.38 -> 61.25 | Hurt |
| Diversify sampled training clips by prioritizing clips from different videos (ablation) | ViCLIP · Kinetics zero-shot action recognition | AVG accuracy: 71.09 -> 74.15; 69.34 -> 72.07; 61.25 -> 64.59 | Helped |
| Filter training clips to the top 30% by UMT-SIM (ablation) | ViCLIP · Kinetics zero-shot action recognition | AVG accuracy: 74.15 -> 75.70; 72.07 -> 73.53; 64.59 -> 66.38 | Helped |
| Include K710 in fine-tuning (ablation) | ViCLIP + InternVid-200M · Kinetics-400 and SomethingSomethingV2 fine-tuned action recognition | top-1 accuracy: 87.9 -> 88.7; 73.6 -> 74.2 | Helped |
| Train on InternVid rather than WebVid10M | ViCLIP · Zero-shot video retrieval across MSR-VTT, LSMDC, DiDeMo, MSVD, and ANet | R@1: The paper reports an average increase of 6.3% in R@1 | Helped |
| Use InternVid-10M-DIV or InternVid-10M-FLT instead of WebVid10M (ablation) | ViCLIP · Zero-shot video retrieval | R@1: gains in R@1 of 14.0% and 17.1% | Helped |
| Add InternVid-Aesthetics-18M to WebVid10M training data | Text-to-video baseline · UCF-101 generation | FVD: 705.25 -> 616.51 | Helped |
| Replace VideoChat's visual encoder with ViCLIP trained on InternVid | VideoChat · Video conversation evaluation | average score: 2.29 to 2.64 | Helped |
| Use multiscale captions instead of VideoChat-generated captions (ablation) | ViCLIP-B trained on InternVid-10M-FLT · MSR-VTT zero-shot retrieval | text-to-video R@1: 33.9 -> 38.6 | Helped |
| Use multiscale captions instead of VideoChat-generated captions (ablation) | ViCLIP-B trained on InternVid-10M-FLT · Kinetics zero-shot action recognition | AVG accuracy: 67.74 -> 71.11; 64.91 -> 68.27; 56.51 -> 59.98 | Helped |
Surprising findings
- ViCLIP trained on the 10M-FLT subset outperforms the model trained on the full 200M set in zero-shot action recognition, which the authors attribute to false negatives from clips of the same video.
- Scaling InternVid from 50M to 200M gives marginal gains in video retrieval, though zero-shot action recognition keeps improving.
Limitations
- Excludes categories such as surveillance, sports, movies and documentaries due to copyright and platform limits, so coverage is incomplete.
- Retrieval scaling conclusions are not definitive because the model is trained only with contrastive loss and no matching head.
- Clip-caption and aesthetic scores are based on automatic models, and the verb counts use a simple NLTK method that the authors acknowledge may be inaccurate.
Experimental setup
Datasets: InternVid WebVid10M Kinetics-400 Kinetics-600 Kinetics-700 MSR-VTT LSMDC DiDeMo MSVD ActivityNet UCF-101 SomethingSomethingV2
Benchmarks: Kinetics-400 Kinetics-600 Kinetics-700 SomethingSomethingV2 MSR-VTT LSMDC DiDeMo MSVD ActivityNet UCF-101
Metrics: top-1 accuracy AVG top-1/top-5 accuracy R@1 FVD FID IS CLIPSIM
Baselines: CLIP (CLIP400M) CLIP (DataComp-1B) EVA-CLIP-L EVA-CLIP-E ViCLIP +WebVid10M CLIP4Clip VideoCrafter VideoFusion VideoChat Video-ChatGPT
Builds on: CLIP EVA-CLIP BLIP-2 Tag2Text UMT
Closest papers by meaning
- Mimir: Improving Video Diffusion Models for Precise Text Understanding (CVPR 2025 · similarity 0.88)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.88)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.88)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.88)
- Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.