Research map / Large language models / Agents, code and math
Code generation benchmarks
Evaluate LLM code generation with evolving, domain-specific, and verification-focused benchmarks.
96 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Agents, code and math, keyword cluster “programs · program synthesis · programming”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 1 tracked benchmark, but the best result has barely moved since 2024.
21 papers in 2023–24 and 69 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 5.8% to 5.5%.
What looks settled
- HumanEval (pass@1): best up to 2024 94.5, best in 2025–26 79.9 from 2 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · DeepDSL: A Compilation-based Domain-Specific Language for Deep Learning (ICLR 2017 · 3 citations)
- 2020 · Program Synthesis with Pragmatic Communication (NeurIPS 2020 · 6 citations)
- 2021 · Latent Execution for Neural Program Synthesis Beyond Domain-Specific Languages (NeurIPS 2021 · 11 citations)
- 2022 · Fault-Aware Neural Code Rankers (NeurIPS 2022 · 13 citations)
- 2023 · Self-Refine: Iterative Refinement with Self-Feedback (NeurIPS 2023 · 350 citations)
- 2024 · Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation (AAAI 2024 · 63 citations)
- 2025 · BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · 22 citations)
- 2026 · NURBGen: High-Fidelity Text-to-CAD Generation Through LLM-Driven NURBS Modeling (AAAI 2026 · 2 citations)
Earliest papers here
- DeepDSL: A Compilation-based Domain-Specific Language for Deep Learning (ICLR 2017 · 3 citations)
- Program Synthesis with Pragmatic Communication (NeurIPS 2020 · 6 citations)
- Latent Execution for Neural Program Synthesis Beyond Domain-Specific Languages (NeurIPS 2021 · 11 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
HumanEval · pass@1 (higher is better)
5 papers, 2023–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 29.28 | CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (ICLR 2023 · 231 citations) |
| 2024 | 94.5 | Divide-and-Conquer Meets Consensus: Unleashing the Power of Functions in Code Generation (NeurIPS 2024 · 1 citation) |
What has been tried
56 papers here report 557 measured changes (a component added, removed or swapped, compared with a baseline): helped 71%, no clear effect 6%, hurt 17%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add a correct same-API usage example to the prompt | RobustAPI · API Misuse Rate | 62.97% -> 38.56% | Helped | Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation (AAAI 2024 · 63 citations) |
| Use 1-shot instead of zero-shot prompting | Dead Code Insertion · Delta entropy (%) | 26.90 -> 21.00 | Hurt | Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscation (AAAI 2025 · 13 citations) |
| Use monologue reasoning rather than few-shot prompting | CRUXEval-I · pass@1 | 39.5 -> 61.8 | Helped | SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning (NeurIPS 2024 · 13 citations) |
| select mutation locations with UCB | MAP-Elites archive · QD score | 395.62 { ± 17.46} -> 341.17 { ± 14.39} | Hurt | GAVEL: Generating Games via Evolution and Language Models (NeurIPS 2024 · 10 citations) |
| Use the simple few-shot prompt instead of zero-shot | CoDI-Eval overall · average accuracy (%) | 80.5 -> 79.33 | Hurt | Benchmarking Large Language Models on Controllable Generation under Diversified Instructions (AAAI 2024 · 8 citations) |
| Add code explanation before refinement rather than requesting refinement directly | MBPP · success refinement rate | 2.58 -> 4.43 | Helped | LeDex: Training LLMs to Better Self-Debug and Explain Code (NeurIPS 2024 · 5 citations) |
| Use 7 or 8 intent-aligned in-context examples rather than 10 | PKU PosterLayout unannotated test · Avg | 0.0677 -> 0.0663; 0.0677 -> 0.0651 | Helped | PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generation (CVPR 2025 · 4 citations) |
| Replace GPT-4's feedback with human feedback | APPS overall · repair success rate | 33.30% -> 52.60%; 1.58× | Helped | Is Self-Repair a Silver Bullet for Code Generation? (ICLR 2024 · 4 citations) |
| Add problem-interpolation examples to GSM8K+MATH training data | GSM8K · accuracy | 77.3 -> 81.7; +4.4 | Helped | MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning (ICLR 2024 · 4 citations) |
| Use GELU instead of ReLU in the overfitting regularization term | SMS Law test scaling-law fit · Huber loss | 0.0095 -> 0.0085 | Helped | MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · 3 citations) |
| Add a reference answer to the judging prompt | LLM-as-a-Judge benchmark · accuracy | 78.3% ± 1.9% -> 83.6% ± 0.4%; 81.3% ± 0.3% -> 82.1% ± 1.3%; 80.7% ± 0.2% -> 81.9% ± 1.6%; 78.7% | Helped | StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · 3 citations) |
| Add MaxSAT-localized sketches to a basic description-and-test-suite prompt | student programs · programs fixed | CodeGemma: 597 (41.7%) -> 682 (47.7%); CodeLlama: 492 (34.4%) -> 573 (40.0%); Gemma: 496 (34.7%) -> 532 (37.2%); Granite | Helped | Counterexample Guided Program Repair Using Zero-Shot Learning and MaxSAT-based Fault Localization (AAAI 2025 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 38 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- HumanEval · 3 of 38 papers
- APPS · 2 of 38 papers
- CodeForces-CoTs · 2 of 38 papers
- AIME 2024 · 2 of 38 papers
- GSM8K · 2 of 38 papers
Baselines
- GPT-4o · 9 of 38 papers
- GPT-4o-mini · 4 of 38 papers
- DeepSeek-V3 · 4 of 38 papers
- GPT-3.5 · 3 of 38 papers
- LLaMA-3-8B · 3 of 38 papers
- GPT-3.5-turbo · 3 of 38 papers
- GPT-4-Turbo · 3 of 38 papers
- CodeGemma · 2 of 38 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 90 papers; from OpenAlex and the papers' affiliations).
- Microsoft · 7 papers
- Tsinghua University · 6 papers
- Peking University · 5 papers
- Alibaba Group · 4 papers
- University of Science and Technology of China · 3 papers
- University of California, San Diego · 3 papers
- Chinese Academy of Sciences · 3 papers
- Hong Kong University of Science and Technology · 3 papers
- Stanford University · 3 papers
- Carnegie Mellon University · 3 papers
Use it
Download BibTeX of 29 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Mathematical and structural reasoning benchmarks (304 papers)
- Inductive and abstract reasoning evaluation (158 papers)
- LLMs for optimization and heuristics (147 papers)
- LLMs for scientific discovery (100 papers)
Elsewhere in the topic
- Agent instruction-following benchmarks (176 papers)
- Long-horizon LLM agents (155 papers)
- Software engineering agents (140 papers)
- LLM-guided program synthesis (93 papers)
- Execution-based code verification (90 papers)
- Retrieval-augmented formal theorem proving (62 papers)
- LLM-driven embodied and driving agents (54 papers)
- Reasoning benchmarks and platforms (47 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).