Research map / Large language models / Agents, code and math
LLM-guided program synthesis
Synthesize programs with LLMs through decomposition, learned abstractions, and per-instance search.
93 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Agents, code and math, keyword cluster “search · optimization · decision”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
22 papers in 2023–24 and 43 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 6.1% to 3.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Mapping Action Language BC to Logic Programs: A Characterization by Postulates (AAAI 2016 · 0 citations)
- 2017 · Programming with a Differentiable Forth Interpreter (ICML 2017 · 92 citations)
- 2018 · DeepProbLog: Neural Probabilistic Logic Programming (NeurIPS 2018 · 111 citations)
- 2019 · Deep Reinforcement Learning for Syntactic Error Repair in Student Programs (AAAI 2019 · 52 citations)
- 2020 · Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension (ICLR 2020 · 72 citations)
- 2021 · GENSYNTH: Synthesizing Datalog Programs without Language Bias (AAAI 2021 · 12 citations)
- 2022 · Compilation of Aggregates in ASP Systems (AAAI 2022 · 9 citations)
- 2023 · Decomposed Prompting: A Modular Approach for Solving Complex Tasks (ICLR 2023 · 99 citations)
- 2024 · Large Language Models as Optimizers (ICLR 2024 · 93 citations)
- 2025 · CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL (ICLR 2025 · 7 citations)
- 2026 · Scaling Agentic Verifier for Competitive Coding (ICML 2026 · 0 citations)
Earliest papers here
- Mapping Action Language BC to Logic Programs: A Characterization by Postulates (AAAI 2016 · 0 citations)
- Programming with a Differentiable Forth Interpreter (ICML 2017 · 92 citations)
- Neural Programming by Example (AAAI 2017 · 18 citations)
What has been tried
65 papers here report 646 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 9%, hurt 18%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use OPRO instead of one-step generation of 50 instructions | GSM8K · test accuracy | 60.8 -> 76.3 | Helped | Large Language Models as Optimizers (ICLR 2024 · 93 citations) |
| Add query-plan CoT generation | BIRD dev · execution accuracy (%) | 57.75 -> 63.62 | Helped | CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL (ICLR 2025 · 7 citations) |
| Increase pseudo-data volume during domain adaptation | ChEBI-20 molecular captioning and molecule generation · overall metrics | Helped | From Artificially Real to Real: Leveraging Pseudo Data from Large Language Models for Low-Resource Molecule Discovery (AAAI 2024 · 5 citations) | |
| Use goal-directed generation rather than undirected mutation | Programming puzzle generation · semantic diversity | Helped | ACES: Generating a Diversity of Challenging Programming Puzzles with Autotelic Generative Models (NeurIPS 2024 · 3 citations) | |
| Add dataset descriptions to program-generation prompts | Text datasets · label model performance | Mixed | The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators (NeurIPS 2024 · 3 citations) | |
| Use Normalized SQL instead of SQL | Spider validation · execution accuracy | 48.7 -> 58.4 | Helped | Enhancing SQL Query Generation with Neurosymbolic Reasoning (AAAI 2025 · 2 citations) |
| Use the compact sharpASP(STD) encoding instead of aspmc+STD or lp2sat+STD | Group 1 instances · BCP time | Helped | Exact ASP Counting with Compact Encodings (AAAI 2024 · 2 citations) | |
| add enumerative search | LOGO · solve rate (mean %) | 21.02 -> 48.95 | Helped | LILO: Learning Interpretable Libraries by Compressing and Documenting Code (ICLR 2024 · 2 citations) |
| use CNN-based scoring instead of Hamming distance | ARC evaluation set (50 tasks) · accuracy | 25/50 -> 25/50 | No clear effect | ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC) (AAAI 2025 · 1 citation) |
| Increase maximum observation subset size from 2 to 3 | LiveCodeBench · pass@k | No clear effect | Planning in Natural Language Improves LLM Search for Code Generation (ICLR 2025 · 1 citation) | |
| remove adaptive search | History · difficulty | 0.440 -> 0.257 | Hurt | AutoBencher: Towards Declarative Benchmark Construction (ICLR 2025 · 1 citation) |
| Add policy feedback through Expert Iteration | ARC evaluation · cumulative solved tasks | 45/400 -> 59/400 | Helped | CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay (ICML 2024 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 28 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ConceptARC · 2 of 28 papers
- LARC · 2 of 28 papers
- ARC-DSL · 2 of 28 papers
- Spider · 2 of 28 papers
- CLUTRR · 2 of 28 papers
- MBPP+ · 2 of 28 papers
Baselines
- DeepSeek-R1 · 3 of 28 papers
- Hamming distance scoring · 2 of 28 papers
- OpenTuner · 2 of 28 papers
- GPT-4.1 · 2 of 28 papers
- o3-mini · 2 of 28 papers
- GPT-4o · 2 of 28 papers
- Gemini-1.5 Pro · 2 of 28 papers
- Chain of Thought · 2 of 28 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 65 papers; from OpenAlex and the papers' affiliations).
- Google · 4 papers
- Massachusetts Institute of Technology · 4 papers
- National University of Singapore · 3 papers
- University of Cambridge · 3 papers
- Meta · 3 papers
- University of California, Berkeley · 3 papers
- Stanford University · 3 papers
- University of Pennsylvania · 3 papers
- Chinese Academy of Sciences · 3 papers
- University of Chinese Academy of Sciences · 3 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Execution-based code verification (90 papers)
- LLM planning with symbolic tools (44 papers)
- Neural program synthesis and transpilation (31 papers)
Elsewhere in the topic
- Mathematical and structural reasoning benchmarks (304 papers)
- Agent instruction-following benchmarks (176 papers)
- Inductive and abstract reasoning evaluation (158 papers)
- Long-horizon LLM agents (155 papers)
- LLMs for optimization and heuristics (147 papers)
- Software engineering agents (140 papers)
- LLMs for scientific discovery (100 papers)
- Code generation benchmarks (96 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).