Research map / Video / Video-language understanding
Encoder-decoder video captioning
Generates video captions with hierarchical, object-aware and reinforcement-learned encoders.
52 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Video-language understanding, keyword cluster “grounding · captioning · sentence”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 5 of 6 tracked benchmarks.
11 papers in 2023–24 and 7 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 4.9% to 1.0%.
Open questions the data points to
- MSR-VTT (BLEU-4): best went from 48.92 (up to 2024) to 60 in 2025 — MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations)
- MSR-VTT (METEOR): best went from 30.16 (up to 2024) to 34.7 in 2025 — MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations)
- MSR-VTT (CIDEr): best went from 55.1 (up to 2024) to 82.9 in 2025 — MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations)
- MSVD (METEOR): best went from 37.7 (up to 2024) to 48.7 in 2025 — MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations)
- MSVD (BLEU@4): best went from 59.2 (up to 2024) to 76.9 in 2025 — MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations)
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (CVPR 2016 · 563 citations)
- 2017 · Video Captioning With Transferred Semantic Attributes (CVPR 2017 · 378 citations)
- 2018 · Reconstruction Network for Video Captioning (CVPR 2018 · 340 citations)
- 2019 · Memory-Attended Recurrent Network for Video Captioning (CVPR 2019 · 249 citations)
- 2020 · Object Relational Graph With Teacher-Recommended Learning for Video Captioning (CVPR 2020 · 328 citations)
- 2021 · Semantic Grouping Network for Video Captioning (AAAI 2021 · 150 citations)
- 2022 · SwinBERT: End-to-End Transformers With Sparse Attention for Video Captioning (CVPR 2022 · 296 citations)
- 2023 · Text With Knowledge Graph Augmented Transformer for Video Captioning (CVPR 2023 · 84 citations)
- 2024 · ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (NeurIPS 2024 · 56 citations)
- 2025 · Progress-Aware Video Frame Captioning (CVPR 2025 · 3 citations)
- 2026 · Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation (ICML 2026 · 0 citations)
Earliest papers here
- Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (CVPR 2016 · 563 citations)
- A Unified Bayesian Model of Scripts, Frames and Language (AAAI 2016 · 21 citations)
- Video Captioning With Transferred Semantic Attributes (CVPR 2017 · 378 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MSR-VTT · BLEU-4 (higher is better)
11 papers, 2018–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 45.01 | Adaptive Feature Abstraction for Translating Video to Text (AAAI 2018 · 26 citations) |
| 2022 | 48.92 | End-to-End Generative Pretraining for Multimodal Video Captioning (CVPR 2022 · 158 citations) |
| 2025 | 60 | MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations) |
MSR-VTT · METEOR (higher is better)
11 papers, 2017–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 25.9 | Top-Down Visual Saliency Guided by Captions (CVPR 2017 · 144 citations) |
| 2018 | 30.16 | Adaptive Feature Abstraction for Translating Video to Text (AAAI 2018 · 26 citations) |
| 2025 | 34.7 | MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations) |
MSR-VTT · CIDEr (higher is better)
11 papers, 2018–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 51.41 | Adaptive Feature Abstraction for Translating Video to Text (AAAI 2018 · 26 citations) |
| 2022 | 55.1 | SwinBERT: End-to-End Transformers With Sparse Attention for Video Captioning (CVPR 2022 · 296 citations) |
| 2025 | 82.9 | MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations) |
MSVD · METEOR (higher is better)
10 papers, 2017–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 33.5 | Video Captioning With Transferred Semantic Attributes (CVPR 2017 · 378 citations) |
| 2019 | 36.2 | Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning (CVPR 2019 · 188 citations) |
| 2020 | 36.9 | Spatio-Temporal Graph for Video Captioning With Knowledge Distillation (CVPR 2020 · 265 citations) |
| 2022 | 37.7 | Hierarchical Modular Network for Video Captioning (CVPR 2022 · 102 citations) |
| 2025 | 48.7 | MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations) |
MSVD test · CIDEr (higher is better)
10 papers, 2017–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 63.5 | Hierarchical Boundary-Aware Neural Encoder for Video Captioning (CVPR 2017 · 141 citations) |
| 2018 | 80.3 | Reconstruction Network for Video Captioning (CVPR 2018 · 340 citations) |
| 2020 | 96.8 | Joint Commonsense and Relation Reasoning for Image and Video Captioning (AAAI 2020 · 47 citations) |
| 2021 | 109.5 | Augmented Partial Mutual Learning with Frame Masking for Video Captioning (AAAI 2021 · 24 citations) |
| 2022 | 120.6 | SwinBERT: End-to-End Transformers With Sparse Attention for Video Captioning (CVPR 2022 · 296 citations) |
| 2024 | 130.96 | Learnability Matters: Active Learning for Video Captioning (NeurIPS 2024 · 6 citations) |
MSVD · BLEU@4 (higher is better)
9 papers, 2017–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 52.8 | Video Captioning With Transferred Semantic Attributes (CVPR 2017 · 378 citations) |
| 2019 | 56.9 | Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning (CVPR 2019 · 188 citations) |
| 2022 | 59.2 | Hierarchical Modular Network for Video Captioning (CVPR 2022 · 102 citations) |
| 2025 | 76.9 | MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations) |
What has been tried
49 papers here report 526 measured changes (a component added, removed or swapped, compared with a baseline): helped 85%, no clear effect 2%, hurt 12%, mixed 1%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add short captions to VQA training data | Video understanding benchmarks · performance | Mixed | ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (NeurIPS 2024 · 56 citations) | |
| Use recursive video and text inputs rather than video-only input | Ego4D-HCap segment descriptions · CIDEr | 40.17 -> 41.74 | Helped | Video ReCap: Recursive Captioning of Hour-Long Videos (CVPR 2024 · 32 citations) |
| Add conceptual queries | MSRVTT · CIDEr | 63.5 -> 64.7 | Helped | Set Prediction Guided by Semantic Concepts for Diverse Video Captioning (AAAI 2024 · 6 citations) |
| Use uncertainty-based selection rather than random sampling | MSVD · AUC CIDEr | .574 -> .583 | Helped | Learnability Matters: Active Learning for Video Captioning (NeurIPS 2024 · 6 citations) |
| Add syntax-level uncertainty rather than using negative likelihood | UCFCAP · ROUGE | 0.354 -> 0.359 | Helped | Hierarchical Multi-Source Uncertainty Aggregation for Interactive Video Captioning (AAAI 2025 · 0 citations) |
| Add the adaptive attention mask to MAMS + SwinBERT | MSVD · METEOR | 41.6 -> 42.1 | Helped | MAMS: Model-Agnostic Module Selection Framework for Video Captioning (AAAI 2025 · 0 citations) |
| Add MAD data to Stage-2 training | MAD-named-eval, context (3 scenes) · CIDEr | 3.52 -> 25.47 | Helped | CALVIN: Improved Contextual Video Captioning via Instruction Tuning (NeurIPS 2024 · 0 citations) |
| use prioritized subgraph extraction with k=1 instead of k=3 | MSR-VTT test · CIDEr | 24.9 -> 26.1 | Helped | Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| Baseline | MSR-VTT test | CIDEr | 46.1 – 49.8 | 2 papers, e.g. CVPR 2020 and CVPR 2021 |
What reviewers will expect
Datasets and baselines named in the 9 papers of this approach from 2024–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MSR-VTT · 5 of 9 papers
- MSVD · 4 of 9 papers
- Ego4D · 2 of 9 papers
- NExT-QA · 2 of 9 papers
- TextCaps · 2 of 9 papers
- Visual Genome · 2 of 9 papers
Baselines
- BLIP-2 · 2 of 9 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 18 papers; from OpenAlex and the papers' affiliations).
- Meta · 3 papers
- University of Chinese Academy of Sciences · 2 papers
- Zhejiang University · 2 papers
Use it
Download BibTeX of 32 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Temporal video grounding (88 papers)
- Dense video captioning (65 papers)
Elsewhere in the topic
- Long video understanding (236 papers)
- Video LLMs with fine-grained understanding (229 papers)
- Event and anomaly reasoning in video (184 papers)
- Video-language pretraining (82 papers)
- Video question answering (64 papers)
- Text-video retrieval (59 papers)
- Egocentric video understanding (46 papers)
- Video moment retrieval (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).