Research map / Large language models / Agents, code and math
LLM planning with symbolic tools
Combine language models with action languages and commonsense knowledge for task planning.
44 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2022–2026) · an approach in Agents, code and math, keyword cluster “search · optimization · decision”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
18 papers in 2023–24 and 23 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 5.0% to 1.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2022 · Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (ICML 2022 · 155 citations)
- 2023 · PAL: Program-aided Language Models (ICML 2023 · 100 citations)
- 2024 · Generalized Planning in PDDL Domains with Pretrained Large Language Models (AAAI 2024 · 71 citations)
- 2025 · PSI: A Benchmark for Human Interpretation and Response in Traffic Interactions (NeurIPS 2025 · 16 citations)
- 2026 · LLMs Lean on Priors, Not Programming Language Semantics (ICML 2026 · 0 citations)
Earliest papers here
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (ICML 2022 · 155 citations)
- TEACh: Task-Driven Embodied Agents That Chat (AAAI 2022 · 100 citations)
- Communicating Natural Programs to Humans and Machines (NeurIPS 2022 · 4 citations)
What has been tried
34 papers here report 335 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 5%, hurt 17%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| remove CoT summarization | Delivery · fraction of evaluation tasks solved | 0.90 -> 0.70 | Hurt | Generalized Planning in PDDL Domains with Pretrained Large Language Models (AAAI 2024 · 71 citations) |
| Add disagreement-based objective reweighting | PSI intent prediction · Acc | 60.30 -> 61.67 | Helped | PSI: A Benchmark for Human Interpretation and Response in Traffic Interactions (NeurIPS 2025 · 16 citations) |
| Apply Soft-Prompt Tuning instead of ICL | TaskLAMA test · Rouge1 F1 | 33.1 -> 38.7 | Helped | TaskLAMA: Probing the Complex Task Understanding of Language Models (AAAI 2024 · 14 citations) |
| Add both planning and summarization language-model modules | real-world websites · success | 10.0 -> 65.0; 20.0 -> 70.0; 10.0 -> 80.0 | Helped | A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (ICLR 2024 · 14 citations) |
| Use a LowInfo seed prompt rather than an informative seed prompt | Human, rat, and fruit fly datasets · normalized likelihood | Hurt | Discovering Symbolic Cognitive Models from Human and Animal Behavior (ICML 2025 · 13 citations) | |
| Replace code rationales with textual rationales | AsDiv, GSM8K, MultiArith, and SVAMP · accuracy | Hurt | When Do Program-of-Thought Works for Reasoning? (AAAI 2024 · 8 citations) | |
| Add Self-Revision with BC+ reasoner feedback | 30 problems · programs executable without syntax errors | 22.3\% (7/30) -> 86.7\% (26/30) | Helped | LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning About Actions (AAAI 2025 · 4 citations) |
| replace random negative decisions with Negative Operations Sampling | GUI-I · critic accuracy | 50.46 -> 69.20 | Helped | Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation (NeurIPS 2025 · 3 citations) |
| add reward R1 (successful compilation) | SVAMP, MultiArith, GSM8K · accuracy (%) | 51.0 -> 56.65; 52.3 -> 59; 34.95 -> 35.2 | Helped | Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning (AAAI 2024 · 3 citations) |
| add task-specific cumulative heuristic g₁,t(n) | ToolBench average · success rate | 65.5 -> 68.5 | Helped | ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Search (ICLR 2024 · 3 citations) |
| Add iterative model evolution and evaluation feedback to a one-shot generated, parameter-optimized HDTwin | Lung Cancer · test MSE | 216±172 -> 4.41±8.07 | Helped | Automatically Learning Hybrid Digital Twins of Dynamical Systems (NeurIPS 2024 · 3 citations) |
| Add context identification without guidelines | WebShop · success rate | 30\text{\%} -> 36\text{\%} | Helped | AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents (NeurIPS 2024 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 19 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- o1-preview · 2 of 19 papers
- self-consistency · 2 of 19 papers
- GPT-4o · 2 of 19 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 41 papers; from OpenAlex and the papers' affiliations).
- IBM · 3 papers
- Google · 3 papers
- Google DeepMind · 3 papers
- Allen Institute for Artificial Intelligence · 3 papers
- University of Cambridge · 2 papers
- National University of Singapore · 2 papers
- Zhejiang University · 2 papers
- Tsinghua University · 2 papers
- Beijing Jiaotong University · 2 papers
- Georgia Institute of Technology · 2 papers
Use it
Download BibTeX of 26 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- LLM-guided program synthesis (93 papers)
- Execution-based code verification (90 papers)
- Neural program synthesis and transpilation (31 papers)
Elsewhere in the topic
- Mathematical and structural reasoning benchmarks (304 papers)
- Agent instruction-following benchmarks (176 papers)
- Inductive and abstract reasoning evaluation (158 papers)
- Long-horizon LLM agents (155 papers)
- LLMs for optimization and heuristics (147 papers)
- Software engineering agents (140 papers)
- LLMs for scientific discovery (100 papers)
- Code generation benchmarks (96 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).