Research map / Video / Video-language understanding
Long video understanding
Handles hour-long videos using memory, key-frame reasoning and training-free or streaming approaches.
236 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Video-language understanding, keyword cluster “video understanding · question · long”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 5 of 6 tracked benchmarks.
25 papers in 2023–24 and 199 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 11.2% to 27.3%.
Open questions the data points to
- MSRVTT-QA (accuracy): best went from 59.3 (up to 2024) to 62.8 in 2025 — DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understanding (CVPR 2025 · 0 citations)
- ActivityNet-QA (accuracy): best went from 50.1 (up to 2024) to 61.8 in 2025 — AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025 · 0 citations)
- NExT-QA (accuracy): best went from 65 (up to 2024) to 81 in 2025 — Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025)
- EgoSchema (accuracy): best went from 37.3 (up to 2024) to 75.2 in 2025 — Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025)
- EgoSchema subset (accuracy): best went from 62.6 (up to 2024) to 75.2 in 2025 — Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025)
- Papers report Video-LLaVA on TGIF-QA (accuracy) anywhere from 41.7 to 70: comparisons against it depend on whose number you take.
- Papers report Video-LLaVA on MSVD-QA (score) anywhere from 3.9 to 71.8: comparisons against it depend on whose number you take.
What looks settled
- MSVD-QA (accuracy): best up to 2024 75.2, best in 2025–26 75.8 from 7 papers, less than 1% better.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · A Dataset and Exploration of Models for Understanding Video Data Through Fill-In-The-Blank Question-Answering (CVPR 2017 · 86 citations)
- 2018 · Multimodal Keyless Attention Fusion for Video Classification (AAAI 2018 · 128 citations)
- 2019 · Long-Term Feature Banks for Detailed Video Understanding (CVPR 2019 · 461 citations)
- 2020 · Violin: A Large-Scale Dataset for Video-and-Language Inference (CVPR 2020 · 60 citations)
- 2021 · Towards Long-Form Video Understanding (CVPR 2021 · 120 citations)
- 2022 · Revisiting the "Video" in Video-Language Understanding (CVPR 2022 · 134 citations)
- 2023 · Selective Structured State-Spaces for Long-Form Video Understanding (CVPR 2023 · 107 citations)
- 2024 · MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (CVPR 2024 · 154 citations)
- 2025 · VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos (CVPR 2025 · 46 citations)
- 2026 · VQAThinker: Exploring Generalizable and Explainable Video Quality Assessment via Reinforcement Learning (AAAI 2026 · 5 citations)
Earliest papers here
- A Dataset and Exploration of Models for Understanding Video Data Through Fill-In-The-Blank Question-Answering (CVPR 2017 · 86 citations)
- Multimodal Keyless Attention Fusion for Video Classification (AAAI 2018 · 128 citations)
- Long-Term Feature Banks for Detailed Video Understanding (CVPR 2019 · 461 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MSRVTT-QA · accuracy (higher is better)
10 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 59.3 | Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · 5 citations) |
| 2025 | 62.8 | DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understanding (CVPR 2025 · 0 citations) |
ActivityNet-QA · accuracy (higher is better)
9 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 50.1 | Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · 5 citations) |
| 2025 | 61.8 | AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025 · 0 citations) |
MSVD-QA · accuracy (higher is better)
9 papers, 2024–2025 · small gains only
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 75.2 | MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (CVPR 2024 · 154 citations) |
| 2025 | 75.8 | Efficient Motion-Aware Video MLLM (CVPR 2025 · 4 citations) |
NExT-QA · accuracy (higher is better)
8 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 65 | Memory Consolidation Enables Long-Context Video Understanding (ICML 2024 · 3 citations) |
| 2025 | 81 | Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025) |
EgoSchema · accuracy (higher is better)
6 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 37.3 | Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · 5 citations) |
| 2025 | 75.2 | Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025) |
EgoSchema subset · accuracy (higher is better)
5 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 62.6 | Memory Consolidation Enables Long-Context Video Understanding (ICML 2024 · 3 citations) |
| 2025 | 75.2 | Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025) |
What has been tried
91 papers here report 1,050 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 4%, hurt 20%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use the memory mechanism instead of removing it | MovieChat-1K global mode · Accuracy | 51.4 -> 67.8 | Helped | MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (CVPR 2024 · 154 citations) |
| add sliding video Q-Former | YouCook2 · SODA_c | +1.0 | Helped | TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · 130 citations) |
| increase GPT-4o max_frames from 1 to 256 | LongVideoBench validation, all durations · accuracy | 41.7 -> 66.7 | Helped | LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · 49 citations) |
| add depth expansion | EgoSchema subset · accuracy | 64.4 -> 66.2; 1.8% | Helped | VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos (CVPR 2025 · 46 citations) |
| Use Sparse Dense Sampling instead of alternative temporal strategies | MeViS valid_u · J&F | 49.9, 50.8, 50.2 -> 51.7 | Helped | One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos (NeurIPS 2024 · 30 citations) |
| add image-based training data | VSI-Bench · accuracy | 32.3 -> 34.6 | Helped | Video-R1: Reinforcing Video Reasoning in MLLMs (NeurIPS 2025 · 27 citations) |
| Add TDA and TKS | ReVOS referring · J &F | 56.8 -> 62.1 | Helped | The Devil is in Temporal Token: High Quality Video Reasoning Segmentation (CVPR 2025 · 20 citations) |
| unify global context frames and local query frames | MeViS (valid) · J &F | 47.2 -> 47.4; 46.2 -> 47.4 | Helped | GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation (CVPR 2025 · 19 citations) |
| Add historical memory to streaming encoding | MovieChat-1K global mode · accuracy | 43.8 -> 90.4 | Helped | Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · 19 citations) |
| Add the retrieval module | EgoSchema subset · accuracy | 57.4% to 60.6% | Helped | DrVideo: Document Retrieval Based Long Video Understanding (CVPR 2025 · 18 citations) |
| Add Text-to-Visual attention to MLP-only prompting | MeViS · J &F | Helped | SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation (CVPR 2025 · 18 citations) | |
| Replace uniform sampling with top-K similarity selection | Video-MME standard · accuracy | +0.25 | Helped | BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · 15 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| Video-LLaVA | TGIF-QA | accuracy | 41.7 – 70 | 3 papers, e.g. CVPR 2025 and CVPR 2025 |
| Video-LLaVA | MSVD-QA | score | 3.9 – 71.8 | 2 papers, e.g. CVPR 2025 and ICML 2025 |
| Video-LLaVA | MSRVTT-QA | score | 3.5 – 59 | 2 papers, e.g. CVPR 2025 and ICML 2025 |
| Video-LLaVA | TGIF-QA | score | 4 – 48.4 | 2 papers, e.g. CVPR 2025 and ICML 2025 |
| Video-LLaVA | ActivityNet-QA | accuracy | 40.7 – 47.08 | 2 papers, e.g. CVPR 2025 and ICML 2025 |
What reviewers will expect
Datasets and baselines named in the 93 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Video-MME · 18 of 93 papers
- NExT-QA · 14 of 93 papers
- EgoSchema · 12 of 93 papers
- LongVideoBench · 10 of 93 papers
- ActivityNet-QA · 10 of 93 papers
- MVBench · 10 of 93 papers
- MLVU · 9 of 93 papers
- LLaVA-Video-178K · 8 of 93 papers
Baselines
- GPT-4o · 20 of 93 papers
- Gemini 1.5 Pro · 16 of 93 papers
- Video-LLaVA · 14 of 93 papers
- VideoChat2 · 11 of 93 papers
- MovieChat · 10 of 93 papers
- Video-ChatGPT · 9 of 93 papers
- VideoLLaMA2 · 8 of 93 papers
- GPT-4V · 7 of 93 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 224 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 7 papers
- Hong Kong University of Science and Technology · 7 papers
- Nanyang Technological University · 7 papers
- Google · 7 papers
- Shanghai AI Laboratory · 6 papers
- National University of Singapore · 6 papers
- Tencent · 6 papers
- Alibaba Group · 6 papers
- Tsinghua University · 6 papers
- Huawei · 5 papers
Use it
Download BibTeX of 32 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Event and anomaly reasoning in video (184 papers)
- Video LLMs with fine-grained understanding (229 papers)
- Video question answering (64 papers)
- Egocentric video understanding (46 papers)
Elsewhere in the topic
- Temporal video grounding (88 papers)
- Video-language pretraining (82 papers)
- Dense video captioning (65 papers)
- Text-video retrieval (59 papers)
- Encoder-decoder video captioning (52 papers)
- Video moment retrieval (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).