Research map / Large language models / Agents, code and math
Inductive and abstract reasoning evaluation
Test inductive, abstract, and symbolic reasoning, including synthesized benchmarks.
158 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Agents, code and math, keyword cluster “programs · program synthesis · programming”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 1 tracked benchmark, but the best result has barely moved since 2024.
34 papers in 2023–24 and 99 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 9.4% to 7.9%.
What looks settled
- MATH (accuracy): best up to 2024 62.25, best in 2025–26 52.2 from 2 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Commonsense Interpretation of Triangle Behavior (AAAI 2016 · 32 citations)
- 2017 · Analogical Chaining with Natural Language Instruction for Commonsense Reasoning (AAAI 2017 · 11 citations)
- 2019 · Analysing Mathematical Reasoning Abilities of Neural Models (ICLR 2019 · 85 citations)
- 2020 · Learning Invariants through Soft Unification (NeurIPS 2020 · 0 citations)
- 2021 · HMS: A Hierarchical Solver with Dependency-Enhanced Understanding for Math Word Problem (AAAI 2021 · 60 citations)
- 2022 · Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiability (AAAI 2022 · 10 citations)
- 2023 · Generalizing Math Word Problem Solvers via Solution Diversification (AAAI 2023 · 7 citations)
- 2024 · Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark (AAAI 2024 · 24 citations)
- 2025 · Efficient Rectification of Neuro-Symbolic Reasoning Inconsistencies by Abductive Reflection (AAAI 2025 · 12 citations)
- 2026 · Speech World Model: Causal State–Action Planning with Explicit Reasoning for Speech (ICLR 2026 · 1 citation)
Earliest papers here
- Commonsense Interpretation of Triangle Behavior (AAAI 2016 · 32 citations)
- Analogical Chaining with Natural Language Instruction for Commonsense Reasoning (AAAI 2017 · 11 citations)
- Coupling Distributed and Symbolic Execution for Natural Language Queries (ICML 2017 · 5 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MATH · accuracy (higher is better)
5 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 62.25 | Decompose, Analyze and Rethink: Solving Intricate Problems with Human-like Reasoning Cycle (NeurIPS 2024 · 3 citations) |
GSM8K · accuracy (higher is better)
4 papers, 2024–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 94.31 | Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving (NeurIPS 2024 · 6 citations) |
What has been tried
83 papers here report 863 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 8%, hurt 20%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use more than one few-shot exemplar | MMLU Physics · accuracy | No clear effect | Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models (ICLR 2024 · 22 citations) | |
| Remove output space shaping | MATH overall · accuracy | 44.6~ -> 40.2~ | Hurt | ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving (ICLR 2024 · 21 citations) |
| increase epsilon from 10^{-2} to 10^{-6} | NYC taxi rides · compression precision | 0.46 (0.06) -> 0.99 (0.01) | Helped | Evaluating the World Model Implicit in a Generative Model (NeurIPS 2024 · 14 citations) |
| Set reflection size hyperparameter C to 0.7 instead of 0.8 | Sudoku · inference accuracy | 97.4 -> 99.1 | Helped | Efficient Rectification of Neuro-Symbolic Reasoning Inconsistencies by Abductive Reflection (AAAI 2025 · 12 citations) |
| add predicate embeddings | ReaSCAN average · accuracy | 96.5 -> 97.5 | Helped | NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization (AAAI 2025 · 8 citations) |
| Remove thought descriptions from SRM input | GSM8K and MATH test · accuracy | GSM8K: 86.20 (+7.73) -> 85.82 (+7.35); MATH: 38.58 (+7.42) -> 39.64 (+8.48) | No clear effect | What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning (AAAI 2025 · 6 citations) |
| use CoMCTS with GPT-4o only rather than direct prediction | Geo3K and GeoQA-Plus · Search Success Rate | 58.2 -> 63.8 | Helped | Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search (NeurIPS 2025 · 6 citations) |
| Use random repository examples instead of skill-matched exemplars | GSM8K · accuracy | 93.00 -> 92.87 | Hurt | Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving (NeurIPS 2024 · 6 citations) |
| Increase hypotheses per iteration from N=1 to N=5 | ACRE · raw accuracy | 77.8 -> 82.5 | Helped | Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR 2024 · 4 citations) |
| Add symbolic computation | MathVista testmini GPS · accuracy | 55.8 -> 52.4; 62.0 -> 57.2 | Helped | GNS: Solving Plane Geometry Problems by Neural-Symbolic Reasoning with Multi-Modal LLMs (AAAI 2025 · 3 citations) |
| add CoT to ANS-only SFT | A2Seek · AP_c | 31.10 -> 38.24 | Helped | A2Seek: Towards Reasoning-Centric Benchmark for Aerial Anomaly Understanding (NeurIPS 2025 · 3 citations) |
| Add NumRel calculation training | OOD counting · accuracy | 0.71 -> 0.94 | Helped | Emergent Communication for Numerical Concepts Generalization (AAAI 2024 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 48 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- GSM8K · 4 of 48 papers
- MATH · 4 of 48 papers
- CLEVR · 2 of 48 papers
- CRUXEval · 2 of 48 papers
- GPQA · 2 of 48 papers
- ReARC · 2 of 48 papers
- MMLU Pro · 2 of 48 papers
Baselines
- GPT-4o · 6 of 48 papers
- DeepSeek-R1 · 3 of 48 papers
- Claude3.5-sonnet · 3 of 48 papers
- self-consistency · 3 of 48 papers
- DeepProbLog · 2 of 48 papers
- Gemini-1.5-Flash · 2 of 48 papers
- chain-of-thought prompting · 2 of 48 papers
- deepseek v3 · 2 of 48 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 133 papers; from OpenAlex and the papers' affiliations).
- Massachusetts Institute of Technology · 6 papers
- Microsoft · 5 papers
- Hong Kong University of Science and Technology · 4 papers
- Amazon · 4 papers
- Google · 4 papers
- Beihang University · 4 papers
- University of Oxford · 3 papers
- University of Science and Technology of China · 3 papers
- Nanjing University · 3 papers
- Tsinghua University · 3 papers
Use it
Download BibTeX of 33 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Mathematical and structural reasoning benchmarks (304 papers)
- LLMs for optimization and heuristics (147 papers)
- Code generation benchmarks (96 papers)
- LLMs for scientific discovery (100 papers)
Elsewhere in the topic
- Agent instruction-following benchmarks (176 papers)
- Long-horizon LLM agents (155 papers)
- Software engineering agents (140 papers)
- LLM-guided program synthesis (93 papers)
- Execution-based code verification (90 papers)
- Retrieval-augmented formal theorem proving (62 papers)
- LLM-driven embodied and driving agents (54 papers)
- Reasoning benchmarks and platforms (47 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).