Research map / Video / Video-language understanding
MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MultiVENT 2.0, a large multilingual event-centric video retrieval benchmark of about 218K news videos and 3,900+ manually written queries, and benchmarks VLMs and single-modality pipelines on it.
Problem
Existing video retrieval datasets use short, English-centric, mostly professionally edited videos with vague caption-style queries and small collections, so they do not test event-centric, multilingual retrieval over large diverse collections.
Approach
Builds a large video collection by merging MultiVENT 1.0 with InternVid videos and writes concise search-style event queries that target information present only in a single modality (visual, speech, embedded text, or description), with graded relevance judgments on top-ranked candidates.
Key results
- On MultiVENT 2.0, the mCLIP vision-only baseline reaches nDCG@10 of 0.303 and R@10 of 0.333, while InternVideo2.0 gets only 0.005 nDCG@10.
- VAST (vision-only) reaches nDCG@10 of 0.116, and LanguageBind reaches 0.324, showing strong VLMs lag on event-centric queries despite strong MSR-VTT scores (e.g., InternVideo2.0 0.851 R@10 on MSR-VTT).
- Description-based mCLIP pipeline achieves nDCG@10 of 0.284 and is best for raw footage (0.307 vs 0.152-0.198 for vision), while speech works best for professional broadcasts (0.318).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use speech transcripts instead of visual content for retrieval | Single-modality pipeline systems on MultiVENT 2.0 professional news broadcasts · Professional news broadcasts | performance: 0.201 -> 0.318 | Helped |
| Use speech transcripts instead of visual content for retrieval | Single-modality pipeline systems on MultiVENT 2.0 true raw footage · True Raw | performance: 0.198 -> 0.021 | Hurt |
| Use embedded OCR text instead of visual content for retrieval | Single-modality pipeline systems on MultiVENT 2.0 true raw footage · True Raw | performance: 0.198 -> 0.006 | Hurt |
| Use human-written descriptions instead of visual content for retrieval | Single-modality pipeline systems on MultiVENT 2.0 true raw footage · True Raw | performance: 0.198 -> 0.307 | Helped |
| Use the query's target modality for retrieval | Single-modality pipeline systems on MultiVENT 2.0 modality-specific queries · MultiVENT 2.0 | performance: 0.331 -> 0.219; 0.225 -> 0.282; 0.232 -> 0.245 | Mixed |
| Use visual content instead of human-written descriptions for retrieval | Single-modality pipeline systems on MultiVENT 2.0 event types · MultiVENT 2.0 | performance: 0.334 vs 0.343; 0.215 vs 0.173; 0.310 vs 0.260; 0.334 vs 0.227; 0.441 vs 0.335; 0.486 vs 0.443; 0.373 vs 0.302; 0.259 vs | Mixed |
| Use a vision-language model on the larger, more diverse event-centric collection rather than MSR-VTT | InternVideo2.0 · Video retrieval benchmark | R@10: 0.851 -> 0.004 | Hurt |
Surprising findings
- Strong VLMs that excel on MSR-VTT (e.g., InternVideo2.0 at 0.851 R@10) nearly fail on MultiVENT 2.0 (0.004 R@10).
- Speech and embedded-text pipelines are almost useless for raw footage, while human-written descriptions remain robust across video types.
Limitations
- Judged@10 only rises to 39% after relevance annotation, so many relevant videos remain unjudged.
- Relies on human-written descriptions as a metadata signal that may not be available for real-time raw footage.
Experimental setup
Datasets: MultiVENT 1.0 InternVid MSR-VTT
Benchmarks: MultiVENT 2.0 MSR-VTT
Metrics: R@10 R@100 MRR mAP nDCG@10
Baselines: mCLIP VAST InternVideo2.0 LanguageBind VALOR ICDAR OCR Whisper
Builds on: MultiVENT 1.0 InternVid MCLIP Whisper
Closest papers by meaning
- Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach (ICLR 2025 · similarity 0.88)
- MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence (NeurIPS 2025 · similarity 0.87)
- Beyond Grounding: Extracting Fine-Grained Event Hierarchies across Modalities (AAAI 2024 · similarity 0.87)
- Localizing Events in Videos with Multimodal Queries (CVPR 2025 · similarity 0.86)
- InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (ICLR 2024 · similarity 0.86)
- Streaming Detection of Queried Event Start (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.