Research map / Video / Video-language understanding
VERIFIED: A Video Corpus Moment Retrieval Benchmark for Fine-Grained Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a fine-grained video corpus moment retrieval (VCMR) benchmark built by VERIFIED, an LLM/LMM caption-generation pipeline with a UMT-based noise evaluator that filters hallucinated captions, yielding Charades-FIG, DiDeMo-FIG, and ActivityNet-FIG.
Problem
Existing VCMR benchmarks use coarse-grained queries that admit many partially matching moments, so models cannot localize precisely with fine-grained queries; manual fine-grained annotation is costly and LLM/LMM captions suffer from hallucination.
Approach
Generate multiple static- and dynamic-enhanced fine-grained captions per moment using image LMM (static attributes), VQA-guided video LMM (dynamics), and LLM rewriting, then filter them with a noise evaluator: UMT fine-tuned with hard-negative contrastive and matching losses on disturbed captions, scoring each caption's confidence and keeping the highest-scoring one.
Key results
- On Charades-FIG, XML trained on FIG data improves VCMR 0.5/r100 from 3.87 (COG) to 9.87 and VR r100 from 30.11 to 51.72.
- User study: over 80% of statics and dynamics captions judged to contain richer details (R_s/R_d between 80 and 100%), with an average accuracy score of 4.46 out of 5.
- Across benchmarks, CONQUER achieves the strongest VCMR results on DiDeMo-FIG (0.5/r1 5.48), while SQuiDNet is best on VR r1 for Charades-FIG and ActivityNet-FIG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train XML on fine-grained FIG rather than coarse-grained COG annotations | XML · Charades-FIG | VCMR 0.5/r5: 0.89 -> 2.63 | Helped |
| Train XML on fine-grained FIG rather than coarse-grained COG annotations | XML · DiDeMo-FIG | VCMR 0.5/r5: 6.08 -> 9.64 | Helped |
| Train XML on fine-grained FIG rather than coarse-grained COG annotations | XML · ActivityNet-FIG | VCMR 0.5/r5: 4.41 -> 7.86 | Helped |
| Select highest-confidence rather than lowest-confidence captions with the evaluator (ablation) | XML · Charades-FIG | VCMR 0.5/r1: 0.67 -> 1.05 | Helped |
| Select highest-confidence rather than lowest-confidence captions with the evaluator (ablation) | XML · DiDeMo-FIG | VCMR 0.5/r1: 2.04 -> 3.19 | Helped |
| Use fine-grained rather than coarse-grained training annotations | XML · Charades-FIG | SVMR 0.5/r1: 24.73 -> 28.20 | Helped |
| Use fine-grained rather than coarse-grained training annotations | XML · DiDeMo-FIG | SVMR 0.5/r1: 17.92 -> 20.53 | Helped |
| Use fine-grained rather than coarse-grained training annotations | XML · ActivityNet-FIG | SVMR 0.5/r1: 24.31 -> 25.23 | Helped |
Surprising findings
- Training the evaluator-filtered captions yields only minor gains for SVMR while substantially improving VR and VCMR.
- SQuiDNet achieves the highest VR accuracy yet is unstable in VCMR and weaker in SVMR, suggesting video-level and moment-level learning should not be entangled.
Limitations
- Gap between captioning-module hallucinations and the synthetic perturbations used to train the evaluator.
- Charades-FIG VCMR and VR performance remains very low because indoor activity videos share similar semantics.
- Pipeline relies on a chain of separate large models rather than an end-to-end captioner.
Experimental setup
Datasets: Charades-STA DiDeMo ActivityNet Captions
Benchmarks: Charades-FIG DiDeMo-FIG ActivityNet-FIG
Metrics: R@K at tIoU 0.5/0.7 (VCMR, SVMR) R@K (VR) vocabulary and POS statistics many-to-many class/instance counts user study ratio and 1-5 accuracy score
Baselines: HERO XML ReLoCLNet CONQUER SQuiDNet
Builds on: UMT XML CONQUER SQuiDNet ReLoCLNet SentenceBERT
Closest papers by meaning
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.90)
- Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language (AAAI 2024 · similarity 0.89)
- Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.88)
- CDTR: Semantic Alignment for Video Moment Retrieval Using Concept Decomposition Transformer (AAAI 2025 · similarity 0.88)
- Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.