Research map / Large language models / Agents, code and math
Mathematical and structural reasoning benchmarks
Benchmark LLM math and structural reasoning, with decomposition-based analysis.
304 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2021–2026) · an approach in Agents, code and math, keyword cluster “programs · program synthesis · programming”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 3 tracked benchmarks, but the best result has barely moved since 2024.
56 papers in 2023–24 and 246 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 15.4% to 19.6%.
What looks settled
- GSM8K (accuracy): best up to 2024 97.35, best in 2025–26 96 from 6 papers, at most 0.2% better, if at all.
- MATH (accuracy): best up to 2024 84.32, best in 2025–26 79.6 from 8 papers, at most 0.2% better, if at all.
- SVAMP (accuracy): best up to 2024 98.16, best in 2025–26 76.3 from 2 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2021 · Measuring Massive Multitask Language Understanding (ICLR 2021 · 325 citations)
- 2022 · Solving Quantitative Reasoning Problems with Language Models (NeurIPS 2022 · 278 citations)
- 2023 · LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models (NeurIPS 2023 · 151 citations)
- 2024 · CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence (NeurIPS 2024 · 40 citations)
- 2025 · GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · 53 citations)
- 2026 · MedOmni-45°: A Safety–Performance Benchmark for Reasoning-Oriented LLMs in Medicine (AAAI 2026 · 4 citations)
Earliest papers here
- Measuring Massive Multitask Language Understanding (ICLR 2021 · 325 citations)
- Solving Quantitative Reasoning Problems with Language Models (NeurIPS 2022 · 278 citations)
- LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models (NeurIPS 2023 · 151 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
GSM8K · accuracy (higher is better)
15 papers, 2022–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 78.5 | Solving Quantitative Reasoning Problems with Language Models (NeurIPS 2022 · 278 citations) |
| 2024 | 97.35 | Self-Guiding Exploration for Combinatorial Problems (NeurIPS 2024 · 3 citations) |
MATH · accuracy (higher is better)
14 papers, 2022–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 50.3 | Solving Quantitative Reasoning Problems with Language Models (NeurIPS 2022 · 278 citations) |
| 2024 | 84.32 | Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification (ICLR 2024 · 17 citations) |
SVAMP · accuracy (higher is better)
5 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 98.16 | Self-Guiding Exploration for Combinatorial Problems (NeurIPS 2024 · 3 citations) |
What has been tried
157 papers here report 1,659 measured changes (a component added, removed or swapped, compared with a baseline): helped 61%, no clear effect 7%, hurt 25%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add one clause to questions | GSM-Symbolic-P1 · accuracy | Hurt | GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · 53 citations) | |
| Add an explicit reasoning prompt | CTI-MCQ · accuracy | 71.00 -> 71.84 | No clear effect | CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence (NeurIPS 2024 · 40 citations) |
| Evaluate code execution with chain-of-thought prompting | LiveCodeBench, code execution · Pass@1 | 39.10 -> 91.00 | Helped | LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · 31 citations) |
| Iteratively retrain the selector between data-collection rounds | MATH · accuracy | No clear effect | Let's Verify Step by Step (ICLR 2024 · 31 citations) | |
| Add PRM to SFT during PPO training | GSM8K and MATH · pass@1 | 77.4 -> 81.7; 35.6 -> 39.9 | Helped | WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (ICLR 2025 · 28 citations) |
| Remove the goal-oriented prompt, retaining RAG only | CausalProbe-H · EM | 0.582 -> 0.575 (LLaMA 2); 0.658 -> 0.655 (LLaMA 3); 0.693 -> 0.678 (GPT 3.5) | Hurt | Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? (NeurIPS 2024 · 24 citations) |
| add single-round DiaS teaching dialogues | Socratic teaching evaluation · Overall | 0.54 -> 0.62 | Helped | SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models (NeurIPS 2024 · 18 citations) |
| Replace code-based self-verification with natural-language self-verification | MATH · accuracy | 69.69 -> 69.29 | Hurt | Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification (ICLR 2024 · 17 citations) |
| Use zero-shot CoT instead of zero-shot prompting | SciBench textbook dataset · accuracy (%) | 33.79 to 28.52 | Hurt | SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · 17 citations) |
| Use validity score p_positive + p_neutral rather than p_positive alone | MR-MATH-invalid solution-level · AUC | 89.7 -> 90.8 | Helped | Evaluating Mathematical Reasoning Beyond Accuracy (AAAI 2025 · 16 citations) |
| Use estimated rather than neutral (0.5) base score for the claim | TruthfulClaim · accuracy | 0.74 -> 0.81 | Helped | Argumentative Large Language Models for Explainable and Contestable Claim Verification (AAAI 2025 · 13 citations) |
| Generate natural-language hypotheses before programs (Full vs Program Only) | 1D-ARC · accuracy (%) | +12.0 (61.1 to 73.1) | Helped | Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · 12 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| MetaMath | GSM8K | accuracy | 66.2 – 72.3 | 2 papers, e.g. ICML 2024 and NeurIPS 2024 |
What reviewers will expect
Datasets and baselines named in the 131 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- GSM8K · 21 of 131 papers
- MATH · 16 of 131 papers
- MMLU · 8 of 131 papers
- HumanEval · 5 of 131 papers
- OpenWebMath · 4 of 131 papers
- SVAMP · 3 of 131 papers
- Omni-MATH · 3 of 131 papers
- AIME 2024 · 3 of 131 papers
Baselines
- GPT-4o · 35 of 131 papers
- GPT-4 · 14 of 131 papers
- GPT-4o-mini · 13 of 131 papers
- Claude 3.5 Sonnet · 11 of 131 papers
- DeepSeek-R1 · 11 of 131 papers
- DeepSeek-V3 · 10 of 131 papers
- GPT-3.5-Turbo · 9 of 131 papers
- Gemini 1.5 Pro · 7 of 131 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 302 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 15 papers
- Microsoft · 14 papers
- Huawei · 11 papers
- Stanford University · 11 papers
- Hong Kong University of Science and Technology · 10 papers
- Princeton University · 10 papers
- University of Science and Technology of China · 9 papers
- University of Washington · 9 papers
- The Chinese University of Hong Kong · 9 papers
- Shanghai Jiao Tong University · 8 papers
Use it
Download BibTeX of 27 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Inductive and abstract reasoning evaluation (158 papers)
- LLMs for optimization and heuristics (147 papers)
- Code generation benchmarks (96 papers)
- LLMs for scientific discovery (100 papers)
Elsewhere in the topic
- Agent instruction-following benchmarks (176 papers)
- Long-horizon LLM agents (155 papers)
- Software engineering agents (140 papers)
- LLM-guided program synthesis (93 papers)
- Execution-based code verification (90 papers)
- Retrieval-augmented formal theorem proving (62 papers)
- LLM-driven embodied and driving agents (54 papers)
- Reasoning benchmarks and platforms (47 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).