Research map / Video / Video-language understanding
VidLA: Video-Language Alignment at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VidLA, a two-tower video-language alignment model with hierarchical temporal attention (multi-scale [MST] tokens) initialized from CLIP, trained on a new ~800M-clip YouTube dataset (YT-VidLA-800M) with LLM-summarized captions and subtitles.
Problem
Video-language alignment at scale is limited by noisy ASR-based video-text data lacking visual grounding and by video architectures that either miss long-range temporal dependencies or are hard to combine with pretrained image-text models.
Approach
A simple CLIP-initialized two-tower model whose vision encoder uses spatially-local temporal attention plus hierarchical multi-scale temporal [MST] tokens, paired with a dataset of clips of varying durations whose subtitles and MLLM-generated captions are summarized by an LLM.
Key results
- VidLA-B/32 reaches 55.6 R@1 text-to-video on MSR-VTT without dual-softmax, 5.5 points above CLIP-ViP (50.1), and VidLA-B/16 reaches 58.0.
- On ActivityNet Captions and DiDeMo, VidLA-B/32* outperforms CLIP-ViP* by 10.1 and 8.4 R@1 points respectively (with dual-softmax).
- Ablation: hierarchical [MST] tokens with spatially-local temporal attention improve MSR-VTT R@1 from 49.1 to 53.5; removing LLM summarization drops Avg from 72.2 to 59.2 without subtitles/captions summarization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VidLA-B/32 instead of CLIPViP | CLIP ViT-B/32 video-text retrieval · MSR-VTT text-to-video | R@1: 50.1 -> 55.6 | Helped |
| Use VidLA-B/32 instead of CLIPViP | CLIP ViT-B/32 video-text retrieval · DiDeMo text-to-video | R@1: 48.6 -> 56.9 | Helped |
| Use VidLA-B/32 instead of CLIPViP | CLIP ViT-B/32 video-text retrieval · ActivityNet Captions text-to-video | R@1: 51.1 -> 61.3 | Helped |
| Use VidLA-B/32 instead of DRL | video-text retrieval · MSVD text-to-video | R@1: 48.3 -> 48.6 | Helped |
| Use VidLA-B/32 instead of DRL | video-text retrieval · VATEX text-to-video | R@1: 63.5 -> 66.5 | Helped |
| Add [MST] tokens (ablation) | VidLA-B/32 · MSR-VTT | R@1: 49.1 -> 49.2 | Helped |
| Add multiple temporal hierarchies to [MST] attention (ablation) | VidLA-B/32 · MSR-VTT | R@1: 49.2 -> 50.0 | Helped |
| Add spatially-local temporal attention (ablation) | VidLA-B/32 · MSR-VTT | R@1: 50.0 -> 53.5 | Helped |
| Use short, medium, and long clips for pretraining instead of short clips only (ablation) | VidLA-B/32 · MSR-VTT | R@1: 51.9 -> 53.5 | Helped |
| Add caption supervision to subtitle supervision (ablation) | VidLA-B/32 pretraining · MSR-VTT | R@1: 48.9 -> 53.5 | Helped |
| Add subtitle supervision to caption supervision (ablation) | VidLA-B/32 pretraining · MSR-VTT | R@1: 50.1 -> 53.5 | Helped |
| Summarize subtitle and caption text (ablation) | VidLA-B/32 pretraining · MSR-VTT | R@1: 36.3 -> 53.5 | Helped |
Limitations
- Relies on proprietary-scale compute (128 A100 GPUs) and a large internal dataset that is not released.
- Classification results trail some baselines (e.g., VideoMAE-v2, UMT on Something-Something-v2), since the model was pretrained only for alignment.
Experimental setup
Datasets: YT-Temporal-1B MSR-VTT DiDeMo ActivityNet Captions MSVD VATEX Kinetics-400 Something-Something-v2
Benchmarks: MSR-VTT DiDeMo ActivityNet Captions MSVD VATEX Kinetics-400 Something-Something-v2
Metrics: R@1 R@5 R@10 Avg MdR MnR top-1 accuracy
Baselines: CLIP4Clip CenterCLIP CLIP2TV CAMoE DRL STAN PIDRo Cap4Video UATVR CLIPViP VINDLU UMT
Builds on: CLIP CLIP4Clip TimeSformer BLIP-2 YT-Temporal-1B
Closest papers by meaning
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024 · similarity 0.90)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.90)
- VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · similarity 0.90)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.89)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.89)
- Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.