Research map / Large language models / Agents, code and math
Execution-based code verification
Verify and debug LLM-generated code using execution feedback, critics, and rigorous tests.
90 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Agents, code and math, keyword cluster “search · optimization · decision”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
36 papers in 2023–24 and 50 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 9.9% to 4.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · DeepFuzz: Automatic Generation of Syntax Valid C Programs for Fuzz Testing (AAAI 2019 · 94 citations)
- 2020 · Structural Language Models of Code (ICML 2020 · 40 citations)
- 2021 · Measuring Coding Challenge Competence With APPS (NeurIPS 2021 · 2 citations)
- 2022 · CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning (NeurIPS 2022)
- 2023 · Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (NeurIPS 2023 · 172 citations)
- 2024 · SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024 · 109 citations)
- 2025 · IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities (ICLR 2025 · 17 citations)
- 2026 · VERINA: Benchmarking Verifiable Code Generation (ICLR 2026 · 2 citations)
Earliest papers here
- DeepFuzz: Automatic Generation of Syntax Valid C Programs for Fuzz Testing (AAAI 2019 · 94 citations)
- Structural Language Models of Code (ICML 2020 · 40 citations)
- Measuring Coding Challenge Competence With APPS (NeurIPS 2021 · 2 citations)
What has been tried
54 papers here report 575 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 5%, hurt 24%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| redo FSM preprocessing for each request instead of reusing the preprocessed state machine for a batch | JSON decoding benchmark · throughput | 2.4× lower | Hurt | SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024 · 109 citations) |
| Use an Oracle retriever with Gorilla instead of no retriever | Torch Hub · accuracy | 59.13 -> 67.20 | Helped | Gorilla: Large Language Model Connected with Massive APIs (NeurIPS 2024 · 93 citations) |
| remove search API and use LLM-generated evidence instead | AmbigNQ · F1 | 74.9 -> 67.3 | Hurt | CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (ICLR 2024 · 57 citations) |
| Add LLM-inferred source specifications | CWE-Bench-Java · #Detected | Helped | IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities (ICLR 2025 · 17 citations) | |
| Add synthesized source/sink extractors instead of directly prompting the LLM | Juliet DBZ, XSS, and OSCI · precision | Helped | LLMDFA: Analyzing Dataflow in Code with Large Language Models (NeurIPS 2024 · 8 citations) | |
| add AST-Aware Segmentation instead of Greedy Segmentation | HumanEval · Pass@1 | 5.2 -> 7.2 | Helped | AST-T5: Structure-Aware Pretraining for Code Generation and Understanding (ICML 2024 · 7 citations) |
| add natural-language rationales during iterative training | MBPP-R · pass@25 | 71.8 -> 79.4 | Helped | NExT: Teaching Large Language Models to Reason about Code Execution (ICML 2024 · 6 citations) |
| use LLM samples directly without search | Tensor · tasks solved | Hurt | HYSYNTH: Context-Free LLM Approximation for Guiding Program Synthesis (NeurIPS 2024 · 5 citations) | |
| generate program summary and invariants together in one step instead of using the two-phase approach | Tensor processing (23 benchmarks) · benchmarks solved | 23 -> 20 | Hurt | Verified Code Transpilation with LLMs (NeurIPS 2024 · 5 citations) |
| Add MLM pretraining | C++ to CUDA · BLEU | 52.12 -> 76.90 | Helped | CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming (NeurIPS 2024 · 4 citations) |
| Add reasoning-trace hallucination remedy | 23 smart contract projects · false positives | 22 -> 10; 54.5% reduction | Helped | Detecting Bugs with Substantial Monetary Consequences by LLM and Rule-based Reasoning (NeurIPS 2024 · 4 citations) |
| Use a clipped prompt containing only the target function instead of the whole file | CoRe Lite pairwise tasks · performance | up to 5.7% | Helped | CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 30 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- HumanEval · 4 of 30 papers
- Devign · 2 of 30 papers
- CodeNet · 2 of 30 papers
- CodeContests · 2 of 30 papers
- MBPP · 2 of 30 papers
Baselines
- GPT-4o · 6 of 30 papers
- DeepSeek-R1 · 4 of 30 papers
- Gemini-2.5-pro · 3 of 30 papers
- GPT-3.5 Turbo · 2 of 30 papers
- Direct prompting · 2 of 30 papers
- OpenAI o3-mini · 2 of 30 papers
- Claude-3.5 Sonnet · 2 of 30 papers
- DeepSeek-V3 · 2 of 30 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 86 papers; from OpenAlex and the papers' affiliations).
- University of California, Berkeley · 8 papers
- Google · 6 papers
- Microsoft · 5 papers
- Meta · 5 papers
- Amazon · 4 papers
- University of Illinois, Urbana Champaign · 4 papers
- University of Edinburgh · 4 papers
- Google DeepMind · 4 papers
- Carnegie Mellon University · 4 papers
- University of Chicago · 4 papers
Use it
Download BibTeX of 31 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- LLM-guided program synthesis (93 papers)
- LLM planning with symbolic tools (44 papers)
- Neural program synthesis and transpilation (31 papers)
Elsewhere in the topic
- Mathematical and structural reasoning benchmarks (304 papers)
- Agent instruction-following benchmarks (176 papers)
- Inductive and abstract reasoning evaluation (158 papers)
- Long-horizon LLM agents (155 papers)
- LLMs for optimization and heuristics (147 papers)
- Software engineering agents (140 papers)
- LLMs for scientific discovery (100 papers)
- Code generation benchmarks (96 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).