Research map / Video / Video-language understanding
ViMU: Benchmarking Video Metaphorical Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ViMU, a hint-free benchmark of 588 videos and 2,352 questions for evaluating video metaphorical/subtext understanding, finding most frontier MLLMs score below 50%.
Problem
Existing video understanding models focus on literal visual content and lack systematic evaluation of implicit, ironic, rhetorical, and social subtext in videos, especially without answer hints.
Approach
A benchmark pairing open-ended interpretation with multiple-choice rhetoric-mechanism, social-value, and evidence-grounding tasks, built via LLM-generated taxonomy annotations and hint-free question generation with iterative LLM refinement and human validation.
Key results
- GPT-5.2 achieves 73.15% on open-ended interpretation and 67.83% on evidence grounding but only 16.55% on rhetoric mechanism identification.
- Most frontier models score below 50% All-Avg; o4-mini reaches the top All-Avg of 46.91%, with Grok-4.1-Fast at 46.28%.
- Models systematically over-predict literal/direct and emotional attitude categories while under-predicting implicit/coded and identity signaling categories.
Surprising findings
- Models that excel on conventional video understanding do not necessarily perform best on metaphorical understanding; GPT-5.2 is top on OE/EG but drops to ~17% on RM.
- Adding taxonomy guidance does not consistently improve global structural fidelity of predicted label geometry.
Limitations
- Benchmark relies on LLM-generated annotations and QA (GPT-5.4) validated by a limited number of human experts (5).
- Contains potentially offensive and harmful content.
- Evaluation of open-ended answers depends on an LLM judge whose identity is not clearly specified.
Experimental setup
Datasets: YouTube, Bilibili, and TikTok videos
Benchmarks: ViMU
Metrics: LLM-judge score (0-9) accuracy Micro-F1 false positive rate shift
Baselines: GPT-5.2 o4-mini Gemini-3-Flash-Preview Grok-4.1-Fast Qwen3.5-27B Qwen3-VL-32B-Instruct Gemma-3-27B-it Claude-3-Haiku GPT-4.1-nano GPT-5.4-mini
Builds on: FunQA v-HUB I-VQA VRR-QA
Closest papers by meaning
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.88)
- MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos (ICLR 2025 · similarity 0.86)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.86)
- SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models (NeurIPS 2025 · similarity 0.86)
- Two Causally Related Needles in a Video Haystack (NeurIPS 2025 · similarity 0.86)
- Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.