Research map / Video / Video-language understanding
Video moment retrieval
Localizes text-described moments in videos with clip trimming, unified retrieval-highlight and generative approaches.
41 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Video-language understanding, keyword cluster “retrieval · text · moment”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
14 papers in 2023–24 and 20 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 6.3% to 2.7%.
Open questions the data points to
- Papers report TGN on DiDeMo test (mIoU) anywhere from 38.62 to 42.97: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · ER3: A Unified Framework for Event Retrieval, Recognition and Recounting (CVPR 2017 · 29 citations)
- 2019 · MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (CVPR 2019 · 357 citations)
- 2021 · Detecting Moments and Highlights in Videos via Natural Language Queries (NeurIPS 2021 · 60 citations)
- 2022 · UMT: Unified Multi-Modal Transformers for Joint Video Moment Retrieval and Highlight Detection (CVPR 2022 · 173 citations)
- 2023 · Query-Dependent Video Representation for Moment Retrieval and Highlight Detection (CVPR 2023 · 157 citations)
- 2024 · Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection (CVPR 2024 · 95 citations)
- 2025 · MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval (CVPR 2025 · 7 citations)
- 2026 · Adaptive Evidential Learning for Temporal-Semantic Robustness in Moment Retrieval (AAAI 2026 · 1 citation)
Earliest papers here
- ER3: A Unified Framework for Event Retrieval, Recognition and Recounting (CVPR 2017 · 29 citations)
- Unified Embedding and Metric Learning for Zero-Exemplar Event Detection (CVPR 2017 · 10 citations)
- MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (CVPR 2019 · 357 citations)
What has been tried
32 papers here report 350 measured changes (a component added, removed or swapped, compared with a baseline): helped 77%, no clear effect 3%, hurt 17%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add a normality visual prompt using match-similarity aggregation | UCF-Crime · AUC | 85.25% -> 87.79% | Helped | Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection (CVPR 2024 · 95 citations) |
| add CIM to the joint framework | QVHighlights val · R1@0.7 | 44.06 -> 51.81 | Helped | Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection (CVPR 2024 · 61 citations) |
| add FW-MESM to the encoder-loss variant | Charades-STA · R1@0.7 | 35.00 -> 36.26 | Helped | Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · 42 citations) |
| Add the task-decoupled unit | QVHighlights val split · Avg. mAP | 41.0 -> 43.24 | Helped | Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection (CVPR 2024 · 19 citations) |
| add query-aware visual enhancement loss Lqav | ActivityNet Captions · R@1, IoU=0.5 | 48.72 -> 52.83 | Helped | Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language (AAAI 2024 · 15 citations) |
| Select highest-confidence rather than lowest-confidence captions with the evaluator | Charades-FIG · VCMR 0.5/r1 | 0.67 -> 1.05 | Helped | VERIFIED: A Video Corpus Moment Retrieval Benchmark for Fine-Grained Video Understanding (NeurIPS 2024 · 8 citations) |
| Add FVE-generated training data | Charades-STA (I3D) · R1@0.5 | 46.19 -> 48.51 | Helped | Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · 6 citations) |
| Add fine alignment to coarse alignment | UCFCrime-AR, Text → Video · R@1 | 24.7 -> 26.0 | Helped | VarCMP: Adapting Cross-Modal Pre-Training Models for Video Anomaly Retrieval (AAAI 2025 · 5 citations) |
| Add MSM-based pre-training | SFX-Moment test · Mean @0.5 | 17.6 -> 23.6 | Helped | D&M: Enriching E-commerce Videos with Sound Effects by Key Moment Detection and SFX Matching (AAAI 2025 · 4 citations) |
| Add VCKA | QVHighlights val · R1@0.5 | 66.39 -> 68.90 | Helped | Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection (AAAI 2025 · 4 citations) |
| Add identity images and face features to the text query and video representation | TVR-IT test-seen · VCG IoU≥0.5 | 7.12 -> 16.86 | Helped | Identity-Text Video Corpus Grounding (AAAI 2025 · 3 citations) |
| fine-tune InternVL on 2,000 human-written pose descriptions | BestShot Full · Top@1 | 32.6 -> 40.5 | Helped | ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries (AAAI 2025 · 1 citation) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| TGN | DiDeMo test | mIoU | 38.62 – 42.97 | 2 papers, e.g. CVPR 2019 and CVPR 2019 |
What reviewers will expect
Datasets and baselines named in the 15 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Charades-STA · 5 of 15 papers
- QVHighlights · 5 of 15 papers
- ActivityNet Captions · 2 of 15 papers
Baselines
- QD-DETR · 5 of 15 papers
- UVCOM · 4 of 15 papers
- M-DETR · 3 of 15 papers
- TR-DETR · 3 of 15 papers
- Moment-DETR+ · 2 of 15 papers
- UniVTG+ · 2 of 15 papers
- R^2-Tuning+ · 2 of 15 papers
- XML · 2 of 15 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 34 papers; from OpenAlex and the papers' affiliations).
- Peking University · 5 papers
- Tsinghua University · 4 papers
- University of Science and Technology of China · 2 papers
- Renmin University of China · 2 papers
- Adobe · 2 papers
- Queen Mary University of London · 2 papers
- Northwestern Polytechnical University · 2 papers
- Sungkyunkwan University · 2 papers
- Meta · 2 papers
- Wuhan University · 2 papers
Use it
Download BibTeX of 31 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Video-language pretraining (82 papers)
- Text-video retrieval (59 papers)
Elsewhere in the topic
- Long video understanding (236 papers)
- Video LLMs with fine-grained understanding (229 papers)
- Event and anomaly reasoning in video (184 papers)
- Temporal video grounding (88 papers)
- Dense video captioning (65 papers)
- Video question answering (64 papers)
- Encoder-decoder video captioning (52 papers)
- Egocentric video understanding (46 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).