Research map / Large language models / Agents, code and math
Measuring Massive Multitask Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale GPT-3 from the smaller variants to the 175 billion-parameter X-Large model | GPT-3 · MMLU | accuracy: 25.9 -> 43.9 | Helped |
| Use UnifiedQA transfer instead of GPT-3 X-Large few-shot | MMLU · MMLU | accuracy: 43.9 -> 48.9 | Helped |
| Use UnifiedQA 3B rather than similarly sized GPT-2 | MMLU · MMLU | accuracy: 32.4% -> 43.7% | Helped |
| Use the UnifiedQA smallest variant rather than RoBERTa-base | MMLU · MMLU | accuracy: 27.9% -> 29.3% | Helped |
| Add up to five few-shot instruction examples to the prompt (ablation) | GPT-3 · MMLU | accuracy | Helped |
| Remove the </s> token from the input (ablation) | UnifiedQA · MMLU | accuracy: several percentage points | Hurt |
| Add legal continued pretraining on approximately 1.6 million case summaries (ablation) | RoBERTa-base · Professional Law | accuracy: 32.8% -> 36.1% | Helped |
| Increase the GPT-3 few-shot prompt from zero-shot to five-shot (ablation) | GPT-3 · MMLU | accuracy | Helped |
Closest papers by meaning
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.85)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.85)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.85)
- Truth Behind the Scene: Designing Evaluations Benchmarks to Assess LLMs’ Task-Specific Understanding over Test-Taking Strategies (AAAI 2025 · similarity 0.84)
- Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process (ICLR 2025 · similarity 0.84)
- RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.