Research map / Video / Video-language understanding
Temporal video grounding
Localizes sentences in videos using local-global interaction, metric learning and unsupervised or scalable grounding.
88 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Video-language understanding, keyword cluster “grounding · captioning · sentence”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 3 of 6 tracked benchmarks.
22 papers in 2023–24 and 37 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 9.9% to 5.1%.
Open questions the data points to
- Charades-STA (R@1, IoU=0.5): best went from 63.98 (up to 2024) to 69.93 in 2025 — Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · 4 citations)
- ActivityNet Captions (R@1 IoU=0.5): best went from 43.79 (up to 2024) to 58.32 in 2025 — Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · 4 citations)
- TACoS (mIoU): best went from 34.63 (up to 2024) to 37.44 in 2025 — Diversifying Query: Region-Guided Transformer for Temporal Sentence Grounding (AAAI 2025 · 7 citations)
What looks settled
- Charades-STA (mIoU): best up to 2024 54.89, best in 2025–26 50.32 from 2 papers, at most 0.2% better, if at all.
No longer reported by papers here: Charades-STA (R@1, IoU=0.7, last in 2022).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos (CVPR 2018 · 99 citations)
- 2019 · Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos (NeurIPS 2019 · 224 citations)
- 2020 · Dense Regression Network for Video Grounding (CVPR 2020 · 303 citations)
- 2021 · Interventional Video Grounding With Dual Contrastive Learning (CVPR 2021 · 151 citations)
- 2022 · Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding (AAAI 2022 · 130 citations)
- 2023 · Weakly Supervised Temporal Sentence Grounding With Uncertainty-Guided Self-Training (CVPR 2023 · 39 citations)
- 2024 · SnAG: Scalable and Accurate Video Grounding (CVPR 2024 · 21 citations)
- 2025 · Number it: Temporal Grounding Videos like Flipping Manga (CVPR 2025 · 18 citations)
- 2026 · TubeRMC: Tube-conditioned Reconstruction with Mutual Constraints for Weakly-supervised Spatio-Temporal Video Grounding (AAAI 2026 · 1 citation)
Earliest papers here
- Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos (CVPR 2018 · 99 citations)
- Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos (NeurIPS 2019 · 224 citations)
- Grounded Video Description (CVPR 2019 · 201 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Charades-STA · mIoU (higher is better)
6 papers, 2020–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 51.38 | Local-Global Video-Text Interactions for Temporal Grounding (CVPR 2020 · 302 citations) |
| 2021 | 54.89 | End-to-end Multi-modal Video Temporal Grounding (NeurIPS 2021 · 21 citations) |
Charades-STA · R@1, IoU=0.5 (higher is better)
6 papers, 2019–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 54.44 | Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos (NeurIPS 2019 · 224 citations) |
| 2021 | 59.7 | Embracing Uncertainty: Decoupling and De-Bias for Robust Temporal Grounding (CVPR 2021 · 55 citations) |
| 2022 | 63.98 | Memory-Guided Semantic Learning Network for Temporal Sentence Grounding (AAAI 2022 · 70 citations) |
| 2025 | 69.93 | Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · 4 citations) |
ActivityNet Captions · R@1 IoU=0.5 (higher is better)
5 papers, 2019–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 36.75 | Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos (NeurIPS 2019 · 224 citations) |
| 2021 | 43.79 | Embracing Uncertainty: Decoupling and De-Bias for Robust Temporal Grounding (CVPR 2021 · 55 citations) |
| 2025 | 58.32 | Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · 4 citations) |
Charades-STA · R@1, IoU=0.7 (higher is better)
5 papers, 2019–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 33.43 | Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos (NeurIPS 2019 · 224 citations) |
| 2021 | 38.52 | Embracing Uncertainty: Decoupling and De-Bias for Robust Temporal Grounding (CVPR 2021 · 55 citations) |
| 2022 | 41.03 | Memory-Guided Semantic Learning Network for Temporal Sentence Grounding (AAAI 2022 · 70 citations) |
Charades-STA · R@0.5 (higher is better)
5 papers, 2020–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 59.46 | Local-Global Video-Text Interactions for Temporal Grounding (CVPR 2020 · 302 citations) |
| 2021 | 63.03 | End-to-end Multi-modal Video Temporal Grounding (NeurIPS 2021 · 21 citations) |
TACoS · mIoU (higher is better)
5 papers, 2021–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 34.63 | Cascaded Prediction Network via Segment Tree for Temporal Video Grounding (CVPR 2021 · 85 citations) |
| 2025 | 37.44 | Diversifying Query: Region-Guided Transformer for Temporal Sentence Grounding (AAAI 2025 · 7 citations) |
What has been tried
70 papers here report 816 measured changes (a component added, removed or swapped, compared with a baseline): helped 79%, no clear effect 3%, hurt 16%, mixed 1%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use late fusion with video-centric training | TACoS · Recall@1, tIoU=0.5 | 44.51 -> 44.86 | Helped | SnAG: Scalable and Accurate Video Grounding (CVPR 2024 · 21 citations) |
| Fine-tune with NumPro-augmented data instead of conventional fine-tuning | ActivityNet · mIoU | 29.0 -> 38.8 | Helped | Number it: Temporal Grounding Videos like Flipping Manga (CVPR 2025 · 18 citations) |
| Add visual and query bias generators | Charades-CD OOD · r1i5 | 43.08 -> 47.20 | Helped | Bias-Conflict Sample Synthesis and Adversarial Removal Debias Strategy for Temporal Sentence Grounding in Video (AAAI 2024 · 18 citations) |
| Replace the single Gaussian with a Gaussian mixture proposal | ActivityNet Captions · R@1, IoU=0.3 | 47.49 -> 59.29 | Helped | Gaussian Mixture Proposals with Pull-Push Learning Scheme to Capture Diverse Events for Weakly Supervised Temporal Video Grounding (AAAI 2024 · 18 citations) |
| add local contrastive learning | Charades-STA · R@0.3 | 65.83 -> 67.26 | Helped | Local-Global Multi-Modal Distillation for Weakly-Supervised Temporal Video Grounding (AAAI 2024 · 16 citations) |
| Add ICG and ICR instance context guidance | HCSTVG-v1 test · m vIoU | 36.52 -> 38.42 | Helped | Context-Guided Spatio-Temporal Video Grounding (CVPR 2024 · 15 citations) |
| Use only the image encoder instead of the dual encoder | GCG test · mIoU | 62.34 -> 60.06 | Hurt | VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (CVPR 2025 · 14 citations) |
| add decoder temporal aggregation | VidSTG test, declarative · m vIoU | 18.07 -> 20.74 | Helped | VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · 13 citations) |
| Add expert salient features to dense sidekick features | Ego4D-NLQ validation · AVG recall | 21.51 -> 23.91 | Helped | DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos (CVPR 2025 · 8 citations) |
| Remove NACS | ActivityNet-CG Test-Trivial and Novel-Comp · R@1 at IoU=0.5 and 0.7 | 51.81 -> 46.33; 33.30 -> 30.71 | Hurt | Exploiting Auxiliary Caption for Video Grounding (AAAI 2024 · 8 citations) |
| Add random boundary shifting and random re-sampling for pseudo-data generation | ActivityNet-Captions · mIoU | 44.46 -> 52.32 | Helped | Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding (CVPR 2024 · 8 citations) |
| add anchor explicit initialization, region-guided attention, and IoU-aware scoring | QVHighlights val · R0.7 | 48.97 -> 52.90 | Helped | Diversifying Query: Region-Guided Transformer for Temporal Sentence Grounding (AAAI 2025 · 7 citations) |
What reviewers will expect
Datasets and baselines named in the 23 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Charades-STA · 7 of 23 papers
- TACoS · 6 of 23 papers
- ActivityNet Captions · 5 of 23 papers
- VidSTG · 5 of 23 papers
- HCSTVG-v1 · 4 of 23 papers
- HCSTVG-v2 · 4 of 23 papers
- QVHighlights · 3 of 23 papers
- Ego4D-NLQ · 3 of 23 papers
Baselines
- 2D-TAN · 6 of 23 papers
- MMN · 5 of 23 papers
- VSLNet · 4 of 23 papers
- CONE · 4 of 23 papers
- SnAG · 4 of 23 papers
- VTimeLLM · 4 of 23 papers
- TubeDETR · 4 of 23 papers
- STCAT · 4 of 23 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 59 papers; from OpenAlex and the papers' affiliations).
- Peking University · 5 papers
- Tsinghua University · 3 papers
- Nanyang Technological University · 3 papers
- National University of Singapore · 3 papers
- Microsoft · 3 papers
- Meituan Inc · 2 papers
- Carnegie Mellon University · 2 papers
- University of Chinese Academy of Sciences · 2 papers
- Harbin Institute of Technology · 2 papers
- University of Oxford · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Encoder-decoder video captioning (52 papers)
- Dense video captioning (65 papers)
Elsewhere in the topic
- Long video understanding (236 papers)
- Video LLMs with fine-grained understanding (229 papers)
- Event and anomaly reasoning in video (184 papers)
- Video-language pretraining (82 papers)
- Video question answering (64 papers)
- Text-video retrieval (59 papers)
- Egocentric video understanding (46 papers)
- Video moment retrieval (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).