Research map / Vision-language and multimodal / Multimodal LLMs
MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MEGA-Bench, a multimodal benchmark of 505 real-world tasks (~8,200 samples) with open-ended output formats, 40+ customized metrics, and multi-dimensional capability breakdowns, evaluating 19 VLMs.
Problem
Existing multimodal benchmarks are narrow, rely mainly on multiple-choice formats, and are expensive and cumbersome to run across many benchmarks, so they fail to measure the breadth of real-world VLM capabilities.
Approach
Scale evaluation to 505 expert-annotated real-world tasks organized by a task taxonomy, embrace diverse output formats (numbers, code, LaTeX, coordinates, JSON, free-form) scored by over 40 rule-based and LLM-assisted metrics, and report fine-grained breakdowns across input type, input count, output format, skill, and application.
Key results
- Claude-3.5-Sonnet (1022) and GPT-4o (0513) lead with overall scores of 54.27 and 54.21, respectively, differing by less than 0.1%.
- Qwen2-VL-72B is the best open-source flagship at 47.70 overall, about 10 points above the next open-source model (InternVL2-Llama3-76B at 37.73).
- Chain-of-thought improves Core scores for proprietary models but 10 of 13 open-source models score worse with CoT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CoT prompting (ablation) | Claude-3.5-Sonnet (1022) · MEGA-Bench Core | score: 49.20 -> 52.59 | Helped |
| Add CoT prompting (ablation) | GPT-4o (0513) · MEGA-Bench Core | score: 52.03 -> 52.65 | Helped |
| Add CoT prompting (ablation) | Qwen2-VL-72B · MEGA-Bench Core | score: 46.41 -> 45.42 | Hurt |
| Add CoT prompting (ablation) | Qwen2-VL-7B · MEGA-Bench Core | score: 34.80 -> 32.93 | Hurt |
| Add CoT prompting (ablation) | Open-source models · MEGA-Bench Core | score: 10 of 13 open-source models get worse results | Mixed |
| Remove the image from the 1-shot demonstration (ablation) | Llama-3.2-11B · MEGA-Bench Core SI | score: 17.34 -> 20.79 | Helped |
| Remove the image from the 1-shot demonstration (ablation) | Llama-3.2-11B · MEGA-Bench Open SI | score: 34.29 -> 38.61 | Helped |
| Use single-image tasks instead of the standard multi-image setting (ablation) | Evaluated models · MEGA-Bench single-image setting | score | Helped |
Surprising findings
- 10 of 13 open-source models get worse Core scores with chain-of-thought prompting, while proprietary models benefit.
- Claude-3.5-Sonnet refuses routine knowledge or commonsense questions, lowering its knowledge and information extraction scores.
Limitations
- Only 15 examples per task on average, so per-task scores have sampling variance.
- Open-ended set relies on GPT-4o as judge, introducing judge dependence.
- Many open-source models are evaluated with a one-shot example and struggle with multi-image inputs and output-format following.
Experimental setup
Datasets: MMMU DocVQA VQA ChartQA InfoVQA
Benchmarks: MEGA-Bench
Metrics: rule-based metric score LLM-judged score (1-10 normalized)
Baselines: MMMU MMT-Bench MMBench MMMU-Pro
Builds on: MMMU MMBench MMT-Bench BIG-bench
Closest papers by meaning
- MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? (ICLR 2025 · similarity 0.90)
- SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.90)
- Task Me Anything (NeurIPS 2024 · similarity 0.90)
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (NeurIPS 2025 · similarity 0.90)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.90)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.