Research map / Large language models / Agents, code and math
LLMs for optimization and heuristics
Use LLMs with evolutionary search to design heuristics and model optimization problems.
147 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Agents, code and math, keyword cluster “programs · program synthesis · programming”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
22 papers in 2023–24 and 123 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 6.1% to 9.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · Argumentation for Explainable Scheduling (AAAI 2019 · 56 citations)
- 2022 · Forecasting Future World Events With Neural Networks (NeurIPS 2022 · 6 citations)
- 2023 · Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering (NeurIPS 2023 · 38 citations)
- 2024 · ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution (NeurIPS 2024 · 53 citations)
- 2025 · Multi-Objective Evolution of Heuristic Using Large Language Model (AAAI 2025 · 26 citations)
- 2026 · EoH-S: Evolution of Heuristic Set Using LLMs for Automated Heuristic Design (AAAI 2026 · 6 citations)
Earliest papers here
- Argumentation for Explainable Scheduling (AAAI 2019 · 56 citations)
- Forecasting Future World Events With Neural Networks (NeurIPS 2022 · 6 citations)
- Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering (NeurIPS 2023 · 38 citations)
What has been tried
60 papers here report 617 measured changes (a component added, removed or swapped, compared with a baseline): helped 75%, no clear effect 7%, hurt 15%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Remove long-term reflections | TSP100 white-box prompting · objective | 8.40 ± 0.02 -> 8.61 ± 0.21 | Hurt | ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution (NeurIPS 2024 · 53 citations) |
| add harmony search to ReEvo | OP · CDI | 5.02 ± 0.13 -> 5.11 ± 0.27 | Helped | HSEvo: Elevating Automatic Heuristic Design with Diversity-Driven Harmony Search and Genetic Algorithm Using LLMs (AAAI 2025 · 20 citations) |
| Add feedback-reflect-refine prompt optimization | SNLI, MNLI, QNLI, RTE, Ethos, Liar, ArSarcasm · F1-score | SNLI: 0.580 -> 0.647; MNLI: 0.746 -> 0.767; QNLI: 0.720 -> 0.793; RTE: 0.659 -> 0.753; Ethos: 0.812 -> 0.963; Liar: 0.57 | Helped | PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refine (AAAI 2024 · 18 citations) |
| add M1, M2, M3, and E2 prompt strategies to E1 | Weibull instances · gap to lower bound | 4.28% -> 2.24% (1k_100); 0.97% -> 0.80% (5k_100); 5.86% -> 2.13% (1k_500) | Helped | Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model (ICML 2024 · 14 citations) |
| shuffle few-shot examples when estimating the discriminative surrogate | Discriminative surrogate evaluation · uncertainty calibration | Helped | Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · 13 citations) | |
| Increase HPO-LLaMA optimization from 1 to 3 rounds | LAMP image classification · accuracy | 0.975±0.028 -> 0.983±0.020 | Helped | AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks (AAAI 2025 · 11 citations) |
| add penetration loss to IDGC | DexGYSNet · CD | 1.276 -> 2.980 | Hurt | Grasp as You Say: Language-guided Dexterous Grasp Generation (NeurIPS 2024 · 11 citations) |
| add discriminator guidance | VerilogEval-human · functional correctness pass@1 | 18.2 -> 20.3 | Helped | BetterV: Controlled Verilog Generation with Discriminative Guidance (ICML 2024 · 9 citations) |
| Remove concept evolution | Feynman Equations · MSE solve rate | Hurt | Symbolic Regression with a Learned Concept Library (NeurIPS 2024 · 9 citations) | |
| replace MCTS planning with single-step Monte Carlo search | Search ablation tasks (average) · accuracy | 0.754 -> 0.635 | Hurt | PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization (ICLR 2024 · 7 citations) |
| add programmer debugging | NL4OPT · accuracy | 72.3% -> 78.7% | Helped | OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models (ICML 2024 · 7 citations) |
| use LLM-generated rather than randomly initialized initial population | Apollo, 24-hour run · time to detect first SV | 39min -> 11min | Helped | An LLM-Empowered Adaptive Evolutionary Algorithm for Multi-Component Deep Learning Systems (AAAI 2025 · 5 citations) |
What reviewers will expect
Datasets and baselines named in the 62 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- NL4OPT · 8 of 62 papers
- IndustryOR · 7 of 62 papers
- ComplexOR · 6 of 62 papers
- NLP4LP · 5 of 62 papers
- MAMO ComplexLP · 4 of 62 papers
- GSM8K · 4 of 62 papers
- TSPLib · 3 of 62 papers
- MAMO EasyLP · 3 of 62 papers
Baselines
- GPT-4o · 9 of 62 papers
- GPT-4 · 5 of 62 papers
- FunSearch · 5 of 62 papers
- EoH · 5 of 62 papers
- OptiMUS · 5 of 62 papers
- DeepSeek-V3 · 5 of 62 papers
- Chain-of-Experts · 4 of 62 papers
- ORLM · 4 of 62 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 145 papers; from OpenAlex and the papers' affiliations).
- Stanford University · 7 papers
- Huawei · 7 papers
- City University of Hong Kong · 6 papers
- University of Cambridge · 5 papers
- Peking University · 5 papers
- University of Science and Technology of China · 5 papers
- Southern University of Science and Technology · 4 papers
- University of Chinese Academy of Sciences · 4 papers
- University of California, Los Angeles · 4 papers
- Chinese Academy of Sciences · 4 papers
Use it
Download BibTeX of 23 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Mathematical and structural reasoning benchmarks (304 papers)
- Inductive and abstract reasoning evaluation (158 papers)
- Code generation benchmarks (96 papers)
- LLMs for scientific discovery (100 papers)
Elsewhere in the topic
- Agent instruction-following benchmarks (176 papers)
- Long-horizon LLM agents (155 papers)
- Software engineering agents (140 papers)
- LLM-guided program synthesis (93 papers)
- Execution-based code verification (90 papers)
- Retrieval-augmented formal theorem proving (62 papers)
- LLM-driven embodied and driving agents (54 papers)
- Reasoning benchmarks and platforms (47 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).