Research map / Video / Video-language understanding
MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMWorld, a multi-discipline, multi-faceted video benchmark (1,910 videos, 6,627 QA pairs) that evaluates MLLMs as world models, finding that the best model, GPT-4V, reaches only 52.3% accuracy.
Problem
Existing video understanding benchmarks do not jointly test domain-spanning knowledge and multi-faceted reasoning (explanation, counterfactuals, future prediction) needed to evaluate MLLMs as world models.
Approach
A video QA benchmark spanning seven disciplines and 69 subdisciplines with human-annotated multi-faceted questions, plus a synthetic audio-only and visual-only subset built by an automated pipeline (GPT-4V-generated QA from keyframes and ASR transcripts) to isolate per-modality perception.
Key results
- GPT-4V achieves the best overall accuracy at 52.3%, with Gemini Pro at 51.0%, while several open-source models score below random chance (26.3%).
- Video-LLaVA-7B outperforms GPT-4V and Gemini Pro on Embodied Tasks (63.17% vs 55.48% and 43.59%) and leads on temporal understanding (34.45%).
- The GPT-4V-based answer judge showed a 4.76% error rate on 189 human-checked examples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o instead of the other evaluated models | MMWorld human-annotated evaluation · MMWorld human-annotated | accuracy: 62.54 | Helped |
| Use Video-LLaVA-7B instead of GPT-4V | MMWorld evaluation · Embodied Tasks | accuracy: 55.48 -> 63.17 | Helped |
| Use Video-LLaVA-7B instead of GPT-4V | MMWorld evaluation · Art & Sports | accuracy: 36.17 -> 35.91 | No clear effect |
| Use Video-LLaVA-7B instead of other evaluated models | MMWorld temporal reasoning evaluation · Temporal Understanding | accuracy: 34.45 | Helped |
| Use Gemini Pro instead of other evaluated models | MMWorld visual-only perception evaluation · MMWorld Synthetic Subset II (Visual) | accuracy: 69.97 | Helped |
| Use Video-Chat instead of ChatUnivi | MMWorld audio-only perception evaluation · MMWorld Synthetic Subset I (Audio) | accuracy: 31.82 -> 38.82 | Helped |
Surprising findings
- Four video-trained MLLMs perform worse than random chance on the benchmark.
- Open-source Video-LLaVA-7B beats proprietary GPT-4V and Gemini Pro on Embodied Tasks and temporal understanding.
Limitations
- Benchmark is not a sufficient testbed for world model evaluation, per the authors' own footnote.
- Human evaluators were primarily non-experts and scored below GPT-4V and Gemini Pro, so human baselines are weak for domain-heavy questions.
- Synthetic QA generated by GPT-4V may carry hallucination risks despite human quality checks.
Experimental setup
Datasets: YouTube-8M SportsQA IKEA Assembly RT-1 Ego4D
Benchmarks: MMWorld
Metrics: accuracy human-validated GPT-4 judge error rate
Baselines: GPT-4V Gemini Pro Video-LLaVA Video-ChatGPT Video-LLaMA Chat-UniVi mPLUG-Owl Otter
Builds on: MMMU Video-ChatGPT UniVTG
Closest papers by meaning
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.91)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.91)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.91)
- MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs (NeurIPS 2025 · similarity 0.90)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.90)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.