Research map / Large language models / Agents, code and math
SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SuperGPQA, a 26,529-question multiple-choice benchmark across 285 graduate disciplines built via a human-LLM collaboration pipeline, and evaluates 51 LLMs on it.
Problem
Existing benchmarks such as MMLU, GPQA, and MMLU-Pro focus on mainstream disciplines and are saturating, leaving long-tail specialized fields (light industry, agriculture, service-oriented disciplines) poorly measured.
Approach
A large-scale human-LLM collaboration annotation pipeline (source screening by experts, crowd-sourced transcription with LLM-assisted distractor generation, and a three-stage quality inspection using SOTA LLM responses) yields a broad, hard, high-option-count benchmark covering 285 subfields.
Key results
- DeepSeek-R1 achieves 61.82% overall accuracy on SuperGPQA (sample-wise), the top score among 51 evaluated models, with o1-2024-12-17 at 60.24%.
- Instruction tuning helps substantially: DeepSeek-V3 scores 47.40 vs 32.14 for DeepSeek-V3-Base, and Qwen2.5-72B-Instruct scores 40.75 vs 34.33 for Qwen2.5-72B.
- Including subfield information in prompts improves Qwen2.5-72B-Instruct from 40.82% to 41.93% zero-shot accuracy, with larger models benefiting more.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| instruction-tune DeepSeek-V3 | DeepSeek-V3 · SuperGPQA overall (sample) | accuracy: 32.14 -> 47.40 | Helped |
| instruction-tune Qwen2.5-72B | Qwen2.5-72B · SuperGPQA overall (sample) | accuracy: 34.33 -> 40.75 | Helped |
| use the newer DeepSeek-R1 model rather than DeepSeek-R1-Zero | DeepSeek-R1 · SuperGPQA overall (sample) | accuracy: 60.24 -> 61.82 | Helped |
| use GPT-4o-2024-11-20 rather than GPT-4o-2024-08-06 | GPT-4o · SuperGPQA overall (sample) | accuracy: 41.64 -> 44.40 | Helped |
| include subfield information in the prompt (ablation) | Qwen2.5-72B-Instruct · SuperGPQA overall | accuracy: 40.82% -> 41.93% | Helped |
| include subfield information in the prompt (ablation) | Qwen2.5-32B-Instruct · SuperGPQA overall | accuracy: 39.13% -> 39.65% | Helped |
| include subfield information in the prompt (ablation) | Qwen2.5-14B-Instruct · SuperGPQA overall | accuracy: 35.36% -> 35.78% | Helped |
| include subfield information in the prompt (ablation) | Qwen2.5-0.5B-Instruct · SuperGPQA overall | accuracy: 11.12% -> 10.62% | Hurt |
| include subfield information in the prompt (ablation) | Qwen2.5-1.5B-Instruct · SuperGPQA overall | accuracy: 18.62% -> 18.09% | No clear effect |
| increase the number of sampled responses using Best of N (ablation) | Qwen2.5-72B-Instruct vs. Doubao-1.5-pro-32k-20250115 · SuperGPQA | accuracy | Mixed |
| use majority voting over independent responses (ablation) | Doubao-1.5-pro-32k-20250115 vs. Qwen2.5-72B-Instruct · SuperGPQA | accuracy: 55-57% | Helped |
Surprising findings
- DeepSeek-R1-Zero (no supervised fine-tuning) performs nearly as well as DeepSeek-R1, beating it in most disciplines.
- Newer o1-mini and o3-mini versions score lower than o1-2024-12-17 outside science and engineering, which the authors suggest may indicate data leakage.
Limitations
- STEM disciplines dominate the question distribution (about 77% in Science, Engineering, and Medicine), so non-STEM coverage is thinner.
- Most questions are multiple-choice, and the authors note that some models may have seen related exercise-website content, raising data leakage concerns.
Experimental setup
Datasets: MMLU GPQA MMLU-Pro MMLU-Redux
Benchmarks: SuperGPQA MMLU GPQA MMLU-Pro
Metrics: accuracy standard deviation coefficient of variation high-low group difference
Baselines: MMLU GPQA MMLU-Pro DeepSeek-R1 o1-2024-12-17 o3-mini DeepSeek-V3 Qwen2.5-72B-Instruct
Builds on: GPQA MMLU-Pro MMLU-Redux
Closest papers by meaning
- Evaluating LLM Reasoning in the Operations Research Domain with ORQA (AAAI 2025 · similarity 0.89)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.89)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.89)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.89)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.89)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.