Research map / Video / Video-language understanding
Video LLMs with fine-grained understanding
Builds video language models for object-level and temporal understanding, with memory and benchmarks.
229 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2021–2026) · an approach in Video-language understanding, keyword cluster “video understanding · question · long”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 5 of 6 tracked benchmarks.
26 papers in 2023–24 and 197 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 11.7% to 27.1%.
Open questions the data points to
- ActivityNet-QA (accuracy): best went from 49.4 (up to 2024) to 57.9 in 2025 — Frame-Voyager: Learning to Query Frames for Video Large Language Models (ICLR 2025 · 0 citations)
- MSVD-QA (accuracy): best went from 75.9 (up to 2024) to 79.8 in 2025 — STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training (CVPR 2025 · 3 citations)
- MVBench (accuracy): best went from 51.1 (up to 2024) to 73.8 in 2025 — PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models (CVPR 2025 · 6 citations)
- NExT-QA (accuracy): best went from 73.2 (up to 2024) to 84.1 in 2025 — Improving LLM Video Understanding with 16 Frames Per Second (ICML 2025)
- EgoSchema (accuracy): best went from 39.7 (up to 2024) to 67.62 in 2025 — LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · 0 citations)
- Papers report Qwen2-VL on VideoMME (accuracy) anywhere from 57.6 to 63.9: comparisons against it depend on whose number you take.
What looks settled
- MSRVTT-QA (accuracy): best up to 2024 69.6, best in 2025–26 65.6 from 5 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2021 · Look Before You Speak: Visually Contextualized Utterances (CVPR 2021 · 54 citations)
- 2022 · IntentVizor: Towards Generic Query Guided Interactive Video Summarization (CVPR 2022 · 38 citations)
- 2023 · Connecting Vision and Language With Video Localized Narratives (CVPR 2023 · 12 citations)
- 2024 · MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · 241 citations)
- 2025 · Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · 149 citations)
- 2026 · StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compression (AAAI 2026 · 7 citations)
Earliest papers here
- Look Before You Speak: Visually Contextualized Utterances (CVPR 2021 · 54 citations)
- MERLOT: Multimodal Neural Script Knowledge Models (NeurIPS 2021 · 54 citations)
- IntentVizor: Towards Generic Query Guided Interactive Video Summarization (CVPR 2022 · 38 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ActivityNet-QA · accuracy (higher is better)
13 papers, 2021–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 38.75 | Look Before You Speak: Visually Contextualized Utterances (CVPR 2021 · 54 citations) |
| 2024 | 49.4 | Slot-VLM: Object-Event Slots for Video-Language Modeling (NeurIPS 2024 · 1 citation) |
| 2025 | 57.9 | Frame-Voyager: Learning to Query Frames for Video Large Language Models (ICLR 2025 · 0 citations) |
MSRVTT-QA · accuracy (higher is better)
12 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 69.6 | Slot-VLM: Object-Event Slots for Video-Language Modeling (NeurIPS 2024 · 1 citation) |
MSVD-QA · accuracy (higher is better)
11 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 75.9 | Slot-VLM: Object-Event Slots for Video-Language Modeling (NeurIPS 2024 · 1 citation) |
| 2025 | 79.8 | STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training (CVPR 2025 · 3 citations) |
MVBench · accuracy (higher is better)
7 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 51.1 | MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · 241 citations) |
| 2025 | 73.8 | PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models (CVPR 2025 · 6 citations) |
NExT-QA · accuracy (higher is better)
7 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 73.2 | Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities (CVPR 2024 · 15 citations) |
| 2025 | 84.1 | Improving LLM Video Understanding with 16 Frames Per Second (ICML 2025) |
EgoSchema · accuracy (higher is better)
6 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 39.7 | SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · 3 citations) |
| 2025 | 67.62 | LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · 0 citations) |
What has been tried
117 papers here report 1,378 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 6%, hurt 22%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use video instruction data instead of image-only instruction data | MVBench · accuracy | 42.1 -> 50.5 | Helped | MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · 241 citations) |
| add subtitles to Gemini 1.5 Pro's frame input | Video-MME overall · accuracy | +6.2% | Helped | Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · 149 citations) |
| Add visual and query memory banks instead of using neither | LVU · top-1 accuracy | 48.3 -> 63.0 | Helped | MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · 80 citations) |
| replace interleaved dialogue training with streaming EOS prediction | Ego4D Narration Stream validation · TimeDiff | 6.47 -> 2.32 | Helped | VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024 · 65 citations) |
| Add quality-aware region selection (QRS) | KVQ · SROCC | 0.832 -> 0.847 | Helped | KVQ: Kwai Video Quality Assessment for Short-form Videos (CVPR 2024 · 45 citations) |
| Use adaptive sampling rather than uniform sampling | LongVideoBench val · accuracy | 58.9 -> 62.7 | Helped | Adaptive Keyframe Sampling for Long Video Understanding (CVPR 2025 · 42 citations) |
| add cross-attention fusion between music and text | Music retrieval test pools · R@1 | 21.40 -> 25.71 | Helped | MuseChat: A Conversational Music Recommendation System for Videos (CVPR 2024 · 35 citations) |
| Replace VST compression with pooling at 16× compression | MLVU-test · score | 41.4 -> 33.7 | Hurt | Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (CVPR 2025 · 33 citations) |
| use autoregressive summary decoding instead of classification-based prediction | LfVS-T · F1 Score | 63.31 -> 68.11 | Helped | Scaling Up Video Summarization Pretraining with Large Language Models (CVPR 2024 · 32 citations) |
| add CAT | MIntRec test-std, twenty-class · ACC | 71.31 -> 73.39 | Helped | Contextual Augmented Global Contrast for Multimodal Intent Recognition (CVPR 2024 · 31 citations) |
| add absolute-time embedding to sequence embedding | YouCook2 · F1 Score | 17.2 -> 17.5 | Helped | VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding (AAAI 2025 · 30 citations) |
| increase video input from 16 to 90 frames | MLVU · M-Avg | 24.2 -> 31.7 | Helped | MLVU: Benchmarking Multi-task Long Video Understanding (CVPR 2025 · 30 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| Qwen2-VL | VideoMME | accuracy | 57.6 – 63.9 | 2 papers, e.g. CVPR 2025 and NeurIPS 2025 |
What reviewers will expect
Datasets and baselines named in the 103 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Video-MME · 17 of 103 papers
- NExT-QA · 15 of 103 papers
- LongVideoBench · 12 of 103 papers
- MLVU · 11 of 103 papers
- ActivityNet-QA · 9 of 103 papers
- ActivityNet Captions · 8 of 103 papers
- Video-ChatGPT · 7 of 103 papers
- MVBench · 7 of 103 papers
Baselines
- GPT-4o · 37 of 103 papers
- Gemini-1.5-Pro · 25 of 103 papers
- Video-LLaVA · 15 of 103 papers
- Video-ChatGPT · 13 of 103 papers
- LLaVA-OneVision · 12 of 103 papers
- Video-LLaMA · 10 of 103 papers
- MovieChat · 9 of 103 papers
- VideoChat2 · 9 of 103 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 223 papers; from OpenAlex and the papers' affiliations).
- Nanjing University · 10 papers
- ByteDance · 9 papers
- Fudan University · 8 papers
- Zhejiang University · 8 papers
- National University of Singapore · 8 papers
- Tsinghua University · 7 papers
- Tencent · 6 papers
- Chinese Academy of Sciences · 6 papers
- The Chinese University of Hong Kong · 6 papers
- University of Science and Technology of China · 6 papers
Use it
Download BibTeX of 23 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Event and anomaly reasoning in video (184 papers)
- Long video understanding (236 papers)
- Video question answering (64 papers)
- Egocentric video understanding (46 papers)
Elsewhere in the topic
- Temporal video grounding (88 papers)
- Video-language pretraining (82 papers)
- Dense video captioning (65 papers)
- Text-video retrieval (59 papers)
- Encoder-decoder video captioning (52 papers)
- Video moment retrieval (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).