Research map / Video / Video-language understanding
Video-language pretraining
Pretrains video-text models with structured alignment, distillation and large-scale transcript data.
82 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Video-language understanding, keyword cluster “retrieval · text · moment”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
43 papers in 2023–24 and 20 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 19.3% to 2.7%.
No longer reported by papers here: MSRVTT-QA (accuracy, last in 2024); MSVD-QA (accuracy, last in 2024); DiDeMo (R@1, last in 2024); MSR-VTT (R@1, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · MSR-VTT: A Large Video Description Dataset for Bridging Video and Language (CVPR 2016 · 1,822 citations)
- 2017 · End-To-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering (CVPR 2017 · 228 citations)
- 2020 · ActBERT: Learning Global-Local Video-Text Representations (CVPR 2020 · 412 citations)
- 2021 · Less Is More: ClipBERT for Video-and-Language Learning via Sparse Sampling (CVPR 2021 · 577 citations)
- 2022 · Advancing High-Resolution Video-Language Representation With Large-Scale Video Transcriptions (CVPR 2022 · 117 citations)
- 2023 · Fine-Tuned CLIP Models Are Efficient Video Learners (CVPR 2023 · 179 citations)
- 2024 · Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · 81 citations)
- 2025 · Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · 4 citations)
- 2026 · VTD-CLIP: Video-to-Text Discretization via Prompting CLIP (AAAI 2026 · 3 citations)
Earliest papers here
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language (CVPR 2016 · 1,822 citations)
- End-To-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering (CVPR 2017 · 228 citations)
- ActBERT: Learning Global-Local Video-Text Representations (CVPR 2020 · 412 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MSRVTT-QA · accuracy (higher is better)
9 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 45.8 | Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations (NeurIPS 2022 · 35 citations) |
MSVD-QA · accuracy (higher is better)
7 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 39.1 | Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (NeurIPS 2022 · 52 citations) |
| 2023 | 54.7 | An Empirical Study of End-to-End Video-Language Transformers With Masked Visual Modeling (CVPR 2023 · 53 citations) |
DiDeMo · R@1 (higher is better)
4 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 28.8 | Advancing High-Resolution Video-Language Representation With Large-Scale Video Transcriptions (CVPR 2022 · 117 citations) |
| 2023 | 61.2 | VindLU: A Recipe for Effective Video-and-Language Pretraining (CVPR 2023 · 59 citations) |
MSR-VTT · R@1 (higher is better)
4 papers, 2020–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 16.3 | ActBERT: Learning Global-Local Video-Text Representations (CVPR 2020 · 412 citations) |
| 2023 | 46.9 | Revisiting Temporal Modeling for CLIP-Based Image-to-Video Knowledge Transferring (CVPR 2023 · 72 citations) |
What has been tried
73 papers here report 880 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 5%, hurt 19%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add text inputs to the student captioning model alongside video input | caption preference user study · preference ratio | +3.0% preference ratio | Helped | Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · 81 citations) |
| Increase pretraining data from InternVid-10M-FLT to InternVid-200M | Kinetics zero-shot action recognition · AVG accuracy | 75.70 -> 71.09; 73.53 -> 69.34; 66.38 -> 61.25 | Hurt | InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (ICLR 2024 · 32 citations) |
| add auxiliary text | Kinetics-400 · top-1 accuracy (%) | 84.2 -> 84.4 | Helped | VicTR: Video-conditioned Text Representations for Activity Recognition (CVPR 2024 · 24 citations) |
| mask 75% of input video tokens during contrastive pre-training | zero-shot video-text retrieval · Recall@1 | an average of 1% absolute difference | Hurt | A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames (CVPR 2024 · 17 citations) |
| Add the video modality to the text-motion model | HumanML3D, All · text-to-motion R@1 | 5.93 -> 6.37 | Helped | Tri-Modal Motion Retrieval by Learning a Joint Embedding Space (CVPR 2024 · 12 citations) |
| Use PaLI-3 frame-level captions as-is rather than alt-text | MSR-VTT zero-shot retrieval · Recall@1 | +2.5 | Helped | Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · 11 citations) |
| Add conceptual descriptions to category names | Something-Something-V2 and Kinetics-400 validation · Top-1 accuracy | 65.7 -> 66.0; 82.9 -> 84.5 | Mixed | Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning (CVPR 2024 · 11 citations) |
| Add [MST] tokens | MSR-VTT · R@1 | 49.1 -> 49.2 | Helped | VidLA: Video-Language Alignment at Scale (CVPR 2024 · 7 citations) |
| Add Text-Guided MultiWay-Sampler instead of using the base fusion setup | MSRQ · accuracy | 44.84 -> 45.54 | Helped | MuLTI: Efficient Video-and-Language Understanding with Text-Guided MultiWay-Sampler and Multiple Choice Modeling (AAAI 2024 · 6 citations) |
| Add stitching-and-matching pre-training to contrastive learning and MLM | MSR-VTT zero-shot retrieval (WebVid1M ablation) · Recall@1 | 21.2 -> 22.4 | Helped | Stitching Segments and Sentences towards Generalization in Video-Text Pre-training (AAAI 2024 · 6 citations) |
| Add hard-negative sampling | Detour video retrieval · R@5 | 16.5 -> 17.6 | Helped | Detours for Navigating Instructional Videos (CVPR 2024 · 6 citations) |
| Use randomly selected captions as negatives instead of hard contrast captions | VideoCon (LLM) Test · ROC-AUC | 59.7 -> 84.6 | Hurt | VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · 6 citations) |
What reviewers will expect
Datasets and baselines named in the 13 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- UCF-101 · 3 of 13 papers
- MSR-VTT · 2 of 13 papers
- ActivityNet · 2 of 13 papers
- Kinetics-400 · 2 of 13 papers
- Kinetics-600 · 2 of 13 papers
- HMDB-51 · 2 of 13 papers
Baselines
- ViFi-CLIP · 2 of 13 papers
- OST · 2 of 13 papers
- XCLIP · 2 of 13 papers
- ActionCLIP · 2 of 13 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 63 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 6 papers
- National University of Singapore · 6 papers
- Alibaba Group · 5 papers
- Chinese Academy of Sciences · 4 papers
- Harbin Institute of Technology · 4 papers
- ByteDance · 4 papers
- Google · 4 papers
- Peng Cheng Laboratory · 3 papers
- Tencent · 3 papers
- The University of Hong Kong · 3 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Text-video retrieval (59 papers)
- Video moment retrieval (41 papers)
Elsewhere in the topic
- Long video understanding (236 papers)
- Video LLMs with fine-grained understanding (229 papers)
- Event and anomaly reasoning in video (184 papers)
- Temporal video grounding (88 papers)
- Dense video captioning (65 papers)
- Video question answering (64 papers)
- Encoder-decoder video captioning (52 papers)
- Egocentric video understanding (46 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).