Video-language understanding: research map
1,681 papers on Video-language understanding in Video, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 3 clusters and 11 approaches. The busiest year so far is 2026.
What it is
Models that understand video together with language: answering questions about videos, retrieving videos from text, finding the moment a sentence describes, captioning, and video LLMs that can follow long videos.
Terms you will see on this page:
- Video LLM
- a multimodal language model that takes video frames as input.
- Temporal grounding
- finding the time span in a video that matches a sentence.
- Dense captioning
- describing every event in a video with its time span.
Start reading with the most cited papers here:
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language (CVPR 2016 · 1,822 citations)
- MovieQA: Understanding Stories in Movies Through Question-Answering (CVPR 2016 · 681 citations)
Within Video, its share grew from 21.6% in 2023–24 to 28.9% in 2025–26 (223 → 728 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Video-language understanding in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Long video understanding rising: 11% → 27% of the topic (25 → 199 papers)
- Video LLMs with fine-grained understanding rising: 12% → 27% of the topic (26 → 197 papers)
- Event and anomaly reasoning in video rising: 13% → 18% of the topic (30 → 130 papers)
- Video-language pretraining falling: 19% → 3% of the topic (43 → 20 papers)
- Temporal video grounding falling: 10% → 5% of the topic (22 → 37 papers)
Papers from 2025–2026 that are already cited most:
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · 149 citations)
- Number it: Temporal Grounding Videos like Flipping Manga (CVPR 2025 · 18 citations)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · 14 citations)
Approaches and key papers
Video LLMs and long video · 1,091 papers
Keywords: video understanding, question, long
Approaches in this cluster:
- Event and anomaly reasoning in video (282 papers)
Reasons about events, anomalies and object state changes in video with datasets and grounded reasoning. - Video LLMs with fine-grained understanding (300 papers)
Builds video language models for object-level and temporal understanding, with memory and benchmarks. - Long video understanding (331 papers)
Handles hour-long videos using memory, key-frame reasoning and training-free or streaming approaches. - Video question answering (110 papers)
Answers questions about videos using memory prompting, sampling, tool use and compositional benchmarks. - Egocentric video understanding (68 papers)
Benchmarks and trains multimodal models for first-person video reasoning and grounded question answering.
Most cited and most cited since 2024:
- MovieQA: Understanding Stories in Movies Through Question-Answering (CVPR 2016 · 681 citations)
- Long-Term Feature Banks for Detailed Video Understanding (CVPR 2019 · 461 citations)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · 241 citations)
- MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (CVPR 2024 · 154 citations)
Temporal grounding and captioning · 315 papers
Keywords: grounding, captioning, sentence
Approaches in this cluster:
- Temporal video grounding (126 papers)
Localizes sentences in videos using local-global interaction, metric learning and unsupervised or scalable grounding. - Encoder-decoder video captioning (73 papers)
Generates video captions with hierarchical, object-aware and reinforcement-learned encoders. - Dense video captioning (116 papers)
Localizes and describes multiple events using joint modeling, memory retrieval and weak supervision.
Most cited and most cited since 2024:
- Lip Reading Sentences in the Wild (CVPR 2017 · 613 citations)
- End-to-End Dense Video Captioning With Masked Transformer (CVPR 2018 · 578 citations)
- VTimeLLM: Empower LLM to Grasp Video Moments (CVPR 2024 · 119 citations)
- ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (NeurIPS 2024 · 56 citations)
Text-video retrieval · 275 papers
Keywords: retrieval, text, moment
Approaches in this cluster:
- Video-language pretraining (123 papers)
Pretrains video-text models with structured alignment, distillation and large-scale transcript data. - Text-video retrieval (103 papers)
Retrieves videos from text using coarse-to-fine, probabilistic and ambiguity-aware representations. - Video moment retrieval (49 papers)
Localizes text-described moments in videos with clip trimming, unified retrieval-highlight and generative approaches.
Most cited and most cited since 2024:
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language (CVPR 2016 · 1,822 citations)
- Less Is More: ClipBERT for Video-and-Language Learning via Sparse Sampling (CVPR 2021 · 577 citations)
- Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection (CVPR 2024 · 95 citations)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · 81 citations)
Related topics in Video
- Video segmentation and tracking (2,850)
- Action recognition (1,325)
- Video generation (2,177)
