Research map / Vision-language and multimodal / Multimodal LLMs

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

CVPR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 355 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMMU, an 11.5K-question college-level multimodal benchmark across six disciplines and 30 subjects, and evaluates 28+ open-source LMMs plus GPT-4V, Gemini, and text-only LLMs, finding that even the best model reaches only about 56% accuracy.

Problem

Existing multimodal benchmarks focus on commonsense or elementary-level knowledge and basic perception, so they do not test expert-level, college-level multimodal understanding and reasoning needed for expert AGI.

Approach

Collects 11.5K college-exam, quiz, and textbook questions that pair heterogeneous images (30 types) with interleaved text and require domain-specific knowledge and deliberate reasoning, and evaluates LMMs zero-shot with rule-based answer extraction.

Key results

• GPT-4V achieves 55.7% test accuracy on MMMU, and Gemini Ultra reaches 59% per the abstract, versus 88.6% for the best human expert on validation.
• Leading open-source models such as BLIP-2 FLAN-T5-XXL and LLaVA-1.5 reach roughly 34% accuracy, while LLaVA-1.6-34B and InternVL-Chat-V1.2 reach 44.7% and 46.2% on test.
• Error analysis of 150 GPT-4V errors attributes 35% to perceptual errors, 29% to lack of knowledge, and 26% to reasoning errors.

Measured effects reported in the paper

Surprising findings

• Adding OCR or LLaVA-generated captions to text-only LLMs yields no notable improvement on MMMU.
• The advantage of stronger models such as GPT-4V nearly disappears on the Hard difficulty split (31.2% for GPT-4V).

Limitations

• Manual curation may introduce biases and the college-level focus may not be sufficient to test Expert AGI.
• Most models are evaluated zero-shot with prompts, and proprietary model results depend on API/playground versions.
• Open-ended questions are only 5.97% of the benchmark, so evaluation is dominated by multiple-choice format.

Experimental setup

Datasets: MMMU

Benchmarks: MMMU

Metrics: micro-averaged accuracy

Baselines: GPT-4V Gemini Ultra BLIP-2 FLAN-T5-XXL LLaVA-1.5 InstructBLIP Qwen-VL CogVLM GPT-4 Text Llama2 7B Vicuna-13B FLAN-T5-XXL

Builds on: MMLU AGIEval ScienceQA

Closest papers by meaning

• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
• MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.88)
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (ICML 2024 · similarity 0.88)
• MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.87)
• RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation (ICML 2025 · similarity 0.87)