Research map / Vision-language and multimodal / Multimodal LLMs
MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMOU, a 20,000-question multiple-choice benchmark over 11,877 long real-world videos testing joint audio-visual reasoning across 13 skills, and shows top models lag far behind humans.
Problem
Existing video and audio benchmarks rarely evaluate joint audio-visual reasoning over long, complex real-world videos; current omni-modal MLLMs' abilities in this setting are largely unmeasured.
Approach
A large expert-annotated omni-modal benchmark on long web videos where each question requires both audio and visual evidence, with 10 options including hard distractors and "None of the above" variants, plus unimodal, cascaded, and open-ended evaluation protocols.
Key results
- Gemini 2.5 Pro achieves 64.2% overall accuracy on the test set, versus 84.3% for human annotators.
- Best open-source model MiniCPM-o 4.5 reaches 46.8%, while Qwen3-Omni-30B-A3B-Thinking reaches only 19.4%.
- Video-only Qwen3-VL-32B reaches 44.0% and audio-only Qwen3-Omni-30B-A3B reaches 35.6%, supporting the need for both modalities.
Surprising findings
- Qwen3-Omni-30B-A3B-Thinking (19.4%) performs far worse than its Instruct variant (46.0%) on the test set.
- Text-only Qwen3-235B (37.5%) outperforms several open-source omni-modal models, and cascaded caption-based pipelines do not match end-to-end Gemini.
Limitations
- Videos come from public web sources, which may cause content bias and train-test leakage in models.
- Multiple-choice evaluation does not fully capture open-ended reasoning.
- Open-ended evaluation relies on LLM judges and is harder to verify.
Experimental setup
Datasets: MMOU
Benchmarks: MMOU
Metrics: accuracy open-ended weighted overall score (correctness, completeness
Baselines: Gemini 2.5 Pro Gemini 2.5 Flash Qwen3-Omni-30B-A3B-Instruct Qwen3-Omni-30B-A3B-Thinking OmniVinci MiniCPM-o Qwen3-VL-32B-Instruct Audio Flamingo 3
Builds on: Video-MME WorldSense OmniBench DailyOmni
Closest papers by meaning
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.94)
- A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.93)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.92)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.92)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.92)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.