Research map / Large language models / Reasoning and chain of thought
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMLU-Pro, a 12,032-question, 10-option, reasoning-focused multi-task benchmark built from filtered MMLU, STEM Website, TheoremQA and SciBench, and evaluates 50+ LLMs with CoT vs direct answering and a 24-prompt robustness study.
Problem
MMLU has saturated (frontier models cluster at 86-87%), relies on knowledge recall with only four options, contains noisy or unanswerable questions, and its scores are highly sensitive to prompt variations, so it poorly discriminates among strong LLMs.
Approach
Build a harder successor to MMLU by filtering out questions answered correctly by most small models, adding reasoning-heavy questions from STEM Website, TheoremQA and SciBench, expanding options from four to ten with GPT-4-Turbo, and running two rounds of expert/LLM-assisted review to remove noise.
Key results
- GPT-4o achieves 72.6% overall accuracy on MMLU-Pro (5-shot CoT), versus about 87% on MMLU, with a 16-33% accuracy drop reported across models.
- CoT improves GPT-4o by 19.1 points on MMLU-Pro versus 1.5 points on MMLU.
- Prompt-induced score sensitivity drops from 4-5% on MMLU to about 2% on MMLU-Pro across 24 prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoT instead of direct answering (ablation) | GPT-4o · MMLU-Pro | accuracy: 53.5 -> 72.6 | Helped |
| Use CoT instead of direct answering (ablation) | GPT-4-Turbo · MMLU-Pro | accuracy: 48.4 -> 63.7 | Helped |
| Use CoT instead of direct answering (ablation) | Phi3-medium-4k-instruct · MMLU-Pro | accuracy: 47.5 -> 55.7 | Helped |
| Use CoT instead of direct answering (ablation) | Llama-3-8B · MMLU-Pro | accuracy: 31.5 -> 35.4 | Helped |
| Use CoT instead of direct answering (ablation) | Gemma-7B · MMLU-Pro | accuracy: 27.0 -> 33.7 | Helped |
| Use CoT instead of direct answering (ablation) | GPT-4-Turbo · MMLU | accuracy: 86.7 -> 86.5 | Hurt |
| Use CoT instead of direct answering (ablation) | Llama-3-8B · MMLU | accuracy: 66.6 -> 62.7 | Hurt |
| Use CoT instead of direct answering (ablation) | Gemma-7B · MMLU | accuracy: 66.0 -> 62.4 | Hurt |
| Expand answer choices from four to ten (ablation) | GPT-4-Turbo on MMLU-Pro · MMLU-Pro | accuracy | No clear effect |
| Use MMLU-Pro instead of MMLU | Language models · MMLU-Pro vs MMLU | accuracy: 16% to 33% | Hurt |
| Evaluate with MMLU-Pro instead of MMLU across varied prompts | Language models · 24 prompt styles | prompt-induced score variability: 4-5% -> 2% | Helped |
Surprising findings
- CoT hurts several models on MMLU (e.g., Llama-3-8B -3.9, Gemma-7B -3.6) while helping them on MMLU-Pro.
- Phi-3 small models perform surprisingly well relative to larger open models, attributed to high-quality educational and code pretraining.
Limitations
- Multiple-choice format cannot capture open-ended generation or depth of comprehension.
- Text-only; excludes multimodal questions requiring images or tables.
- Some remaining noise: expert review removed flawed items, but answer extraction and annotation errors still occur (e.g., 2% annotation errors in GPT-4o error analysis).
Experimental setup
Datasets: MMLU STEM Website TheoremQA SciBench
Benchmarks: MMLU-Pro MMLU
Metrics: accuracy score range across prompts CoT minus direct answer accuracy
Baselines: MMLU GPT-4o GPT-4-Turbo Claude-3-Opus Gemini-1.5-Pro Llama-3-70B-Instruct
Builds on: MMLU TheoremQA SciBench Chain-of-Thought prompting
Closest papers by meaning
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.89)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.89)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.88)
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (ICML 2025 · similarity 0.88)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.88)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.