Research map / Large language models / Agents, code and math
Agent instruction-following benchmarks
Evaluate and improve LLM agents' instruction following and decision-making.
176 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in Agents, code and math, keyword cluster “evaluating · capabilities · bench”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
34 papers in 2023–24 and 142 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 9.4% to 11.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change (NeurIPS 2023 · 46 citations)
- 2024 · LMDrive: Closed-Loop End-to-End Driving with Large Language Models (CVPR 2024 · 167 citations)
- 2025 · Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers (ICLR 2025 · 44 citations)
- 2026 · LLMs Get Lost In Multi-Turn Conversation (ICLR 2026 · 10 citations)
Earliest papers here
- PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change (NeurIPS 2023 · 46 citations)
- Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (NeurIPS 2023 · 33 citations)
- ReAct: Synergizing Reasoning and Acting in Language Models (ICLR 2023)
What has been tried
88 papers here report 900 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 7%, hurt 21%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add Q-Former for visual-token reduction | LangAuto · DS | 31.7±3.5 -> 36.2±2.3 | Helped | LMDrive: Closed-Loop End-to-End Driving with Large Language Models (CVPR 2024 · 167 citations) |
| Add target-task demonstrations when adapting transferred insights | FEVER · success rate (%) | 65 -> \textbf{70} | Helped | ExpeL: LLM Agents Are Experiential Learners (AAAI 2024 · 123 citations) |
| add MedRAG | All datasets · average accuracy (%) | 71.8 -> 75.2 | Helped | MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · 58 citations) |
| Use a Perfect Say model instead of Vicuna | Ravens Tower of Hanoi · planning success | 50 -> 92 | Helped | SayCanPay: Heuristic Planning with Large Language Models Using Learnable Domain Knowledge (AAAI 2024 · 33 citations) |
| Use GPT-3.5 Turbo instead of GPT-4 as tool maker | Logical Deduction (5) · tool generation success | 3/5 -> 0/5 | Hurt | Large Language Models as Tool Makers (ICLR 2024 · 20 citations) |
| use all BEAM scene tokens instead of BEV and map tokens | nuScenes validation · Traj L2 Aveall (m) | 0.56 -> 0.52 | Helped | Distilling Multi-modal Large Language Models for Autonomous Driving (CVPR 2025 · 19 citations) |
| add warm-up training | Seen Toolset · Parameter Correctness | 70.16 -> 77.72 | Helped | Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum (AAAI 2024 · 16 citations) |
| add ground-truth information to feedback generation | MINT reasoning and code generation · feedback gain | +3.43; +8.82 | Helped | MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · 16 citations) |
| add dense grounding tokens | 3D-POPE Random · Precision | +1.38 | Helped | 3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination (CVPR 2025 · 11 citations) |
| Apply conformal prediction filtering to schema candidates | Schema set candidates · number of combinations | 31483 -> 1051 | Helped | Planning in the Dark: LLM-Symbolic Planning Pipeline Without Experts (AAAI 2025 · 8 citations) |
| Use iterative EW-guided P&D Tree refinement with multiple candidates instead of single-candidate P&D Chain | 10-domain average · tasks solved | 0.34 -> 0.66 | Helped | Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models (NeurIPS 2024 · 8 citations) |
| remove the relevance checker | DiscoveryBench · Type I error | 0.103\std{0.020} -> 0.134\std{0.021} | Hurt | Automated Hypothesis Validation with Agentic Sequential Falsifications (ICML 2025 · 7 citations) |
What reviewers will expect
Datasets and baselines named in the 80 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- WebShop · 7 of 80 papers
- ALFWorld · 4 of 80 papers
- ToolBench · 3 of 80 papers
- Blocksworld · 3 of 80 papers
- MATH · 2 of 80 papers
- Yelp · 2 of 80 papers
- ROCStories · 2 of 80 papers
- BabyAI-Text · 2 of 80 papers
Baselines
- GPT-4o · 10 of 80 papers
- CoT · 6 of 80 papers
- ReAct · 6 of 80 papers
- GPT-4o-mini · 6 of 80 papers
- Reflexion · 5 of 80 papers
- GPT-4 · 5 of 80 papers
- GPT-3.5 · 5 of 80 papers
- LATS · 4 of 80 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 176 papers; from OpenAlex and the papers' affiliations).
- Google · 8 papers
- Microsoft · 7 papers
- Massachusetts Institute of Technology · 6 papers
- Tsinghua University · 5 papers
- University of Science and Technology of China · 5 papers
- Shanghai Jiao Tong University · 5 papers
- Alibaba Group · 4 papers
- Stanford University · 4 papers
- University of British Columbia · 4 papers
- ByteDance · 4 papers
Use it
Download BibTeX of 22 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Long-horizon LLM agents (155 papers)
- Software engineering agents (140 papers)
- Reasoning benchmarks and platforms (47 papers)
- LLM-driven embodied and driving agents (54 papers)
Elsewhere in the topic
- Mathematical and structural reasoning benchmarks (304 papers)
- Inductive and abstract reasoning evaluation (158 papers)
- LLMs for optimization and heuristics (147 papers)
- LLMs for scientific discovery (100 papers)
- Code generation benchmarks (96 papers)
- LLM-guided program synthesis (93 papers)
- Execution-based code verification (90 papers)
- Retrieval-augmented formal theorem proving (62 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).