Research map / Video / Video-language understanding
Distilling Vision-Language Models on Millions of Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the image-language model PaLI-3 to video in two stages (visual encoder fine-tuning, then LLM instruction tuning) and uses the adapted model to generate pseudo-captions for millions of web videos, which train a CLIP-style dual encoder.
Problem
Video-text data is scarce and human annotation is costly, while image-based VLM captions lose temporal information and alt-text for videos is noisy and scales poorly.
Approach
Adapt an image VLM to video in two decoupled stages: fine-tune only the visual encoder on short video captions with the language model frozen, then fine-tune only the language model on LLM-generated causal/temporal instruction data with the visual encoder frozen; use the result to caption millions of videos as distilled pseudo-captions.
Key results
- Zero-shot MSR-VTT text-to-video Recall@1 reaches 48.4% with the dual encoder trained on VideoCC++InternVid+, versus 42.4% for the prior ViCLIP (InternVid).
- The adapted video-language model reaches 29.5 WUPS on open-ended NExT-QA, 2.8 points above Flamingo-80B with about 1/16 the parameters (5B).
- Training the dual encoder on VideoCC+ (pseudo-captions) gives 48.2% R@1 on MSR-VTT, 11.2 points above the alt-text baseline (37.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace original VideoCC alt-text with captions generated by the proposed pipeline | CLIP (ViT-st-L) dual-encoder · MSR-VTT zero-shot retrieval | Recall@1: 37.0 -> 48.2; +11.2 | Helped |
| Replace original InternVid captions with captions generated by the proposed pipeline | CLIP (ViT-st-L) dual-encoder · MSR-VTT zero-shot retrieval | Recall@1: 42.5 -> 46.3; +3.8 | Helped |
| Use the adapted video-language model | PaLI-3 · NExT-OE-QA zero-shot QA | WUPS: 26.7 -> 29.5; 2.8% | Helped |
| Use PaLI-3 frame-level captions as-is rather than alt-text (ablation) | CLIP (ViT-st-B) dual-encoder · MSR-VTT zero-shot retrieval | Recall@1: +2.5 | Helped |
| Summarize multiple frame captions with an LLM (ablation) | CLIP (ViT-st-B) dual-encoder · MSR-VTT zero-shot retrieval | Recall@1: 39.5 -> 39.8; +0.3% | Helped |
| Add visual adaptation (ablation) | PaLI-3 caption generation and CLIP dual-encoder · MSR-VTT zero-shot retrieval | Recall@1: 39.5 -> 41.7; +2.2 | Helped |
| Generate multiple captions independently with nucleus sampling (ablation) | CLIP dual-encoder · MSR-VTT zero-shot retrieval | Recall@1: +1.9% | Helped |
| Add instruction-based language adaptation (ablation) | PaLI-3 caption generation and CLIP dual-encoder · MSR-VTT zero-shot retrieval | Recall@1: +0.7% | Helped |
| Fine-tune the visual encoder and freeze the language model instead of freezing the visual encoder and fine-tuning the language model (ablation) | PaLI-3 · S-MiT video captioning | CIDEr: 41.2 -> 42.3 | Helped |
| Fine-tune both the visual encoder and language model (ablation) | PaLI-3 · S-MiT video captioning | CIDEr: 42.3 -> 40.3 | Hurt |
| Add self-training with VideoCC pseudo-captions (ablation) | PaLI-3 · S-MiT video captioning | CIDEr: 42.3 -> 43.5; +1.2 | Helped |
| Use LLaVA 1.0 instruction data for language adaptation (ablation) | PaLI-3 · MSR-VTT zero-shot video captioning | CIDEr: 21.3 -> 16.9 | Hurt |
Surprising findings
- Adding LLaVA 1.0 instruction data lowered MSR-VTT captioning CIDEr by 4.4 relative to no instruction tuning.
- Adding reasoning-style instruction data improved captioning but cut zero-shot QA WUPS to 5.0; restoring QA required adding short-answer QA pairs.
Limitations
- Relies on PaLI-3 and PaLM 2, which are not publicly released, limiting reproducibility.
- Adapted model captions are shown to be vulnerable to hallucination when detailed, and LLM summarization of frame captions fails to recover lost temporal information.
Experimental setup
Datasets: S-MiT VidLN VideoCC InternVid MSR-VTT VATEX Kinetics-600 ActivityNet-Captions ActivityNet-QA NExT-QA MSR-VTT QA Oops-QA
Benchmarks: MSR-VTT VATEX Kinetics-600 ActivityNet-Captions MSR-VTT QA ActivityNet QA NExT-OE-QA
Metrics: Recall@1 CIDEr accuracy WUPS top-1 accuracy
Baselines: PaLI-3 Flamingo-80B FrozenBiLM DeCap ViCLIP InternVideo CLIP4Clip VideoCC alt-text InternVid captions
Builds on: PaLI-3 CLIP LLaVA NExT-QA VideoCLIP
Closest papers by meaning
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.90)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.90)
- VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · similarity 0.90)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.89)
- ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.