Research map / Large language models / Agents, code and math
LLM-driven embodied and driving agents
Use language models for autonomous driving, navigation, and manipulation instructions.
54 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2020–2026) · an approach in Agents, code and math, keyword cluster “evaluating · capabilities · bench”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
15 papers in 2023–24 and 37 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 4.1% to 2.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2020 · Narrative Planning Model Acquisition from Text Summaries and Descriptions (AAAI 2020 · 8 citations)
- 2022 · pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models (NeurIPS 2022 · 30 citations)
- 2023 · ChessGPT: Bridging Policy Learning and Language Modeling (NeurIPS 2023 · 9 citations)
- 2024 · LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs (CVPR 2024 · 48 citations)
- 2025 · Agent4Edu: Generating Learner Response Data by Generative Agents for Intelligent Education Systems (AAAI 2025 · 21 citations)
- 2026 · Driving with Regulation: Trustworthy and Interpretable Decision-Making for Autonomous Driving with Retrieval-Augmented Reasoning (AAAI 2026 · 3 citations)
Earliest papers here
- Narrative Planning Model Acquisition from Text Summaries and Descriptions (AAAI 2020 · 8 citations)
- pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models (NeurIPS 2022 · 30 citations)
- ChessGPT: Bridging Policy Learning and Language Modeling (NeurIPS 2023 · 9 citations)
What has been tried
35 papers here report 367 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 7%, hurt 20%, mixed 1%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add three human-written code exemplars (0-shot to 3-shot) | LaMPilot-Bench · Completion | 20.4% -> 63.3% (+42.9) | Helped | LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs (CVPR 2024 · 48 citations) |
| increase few-shot experiences from 3-shot to 5-shot | lane-4-density-2 · median success steps (SS) | 13 -> 30 | Helped | DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models (ICLR 2024 · 39 citations) |
| Remove profile, memory, memory enhancement, memory forgetting, or reflection | Learner response prediction on EduData · ACC | Hurt | Agent4Edu: Generating Learner Response Data by Generative Agents for Intelligent Education Systems (AAAI 2025 · 21 citations) | |
| Add local reflection | GQA · accuracy | 52.0 -> 54.6 | Helped | CLOVA: A Closed-LOop Visual Assistant with Tool Usage and Update (CVPR 2024 · 12 citations) |
| Add the system prompt | QuickDraw 50 categories · Top-1 accuracy | 0.20 -> 0.23 | Helped | SketchAgent: Language-Driven Sequential Sketch Generation (CVPR 2025 · 10 citations) |
| add Town05 experience to the memory bank and use reflection data | Town05 Short · DS | 75.73±1.36 -> 83.11±0.28 | Helped | Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving (NeurIPS 2024 · 10 citations) |
| pretrain on 70 rather than 50 GPT4-generated simulation tasks | Real-world 12-task evaluation · success rate | 46.7% -> 62.5% | Helped | GenSim: Generating Robotic Simulation Tasks via Large Language Models (ICLR 2024 · 8 citations) |
| add scene representation encoding | R2R val unseen · CIDEr | 0.449 -> 0.465 | Helped | Scene Map-based Prompt Tuning for Navigation Instruction Generation (CVPR 2025 · 6 citations) |
| Add interaction codes (IC) | WOMD · mADE | 1.205 -> 1.067 | Helped | Language-Driven Interactive Traffic Trajectory Generation (NeurIPS 2024 · 5 citations) |
| Add Adaptive Weights to the controller | ScenePlan PartNet · Success Rate (%) | 21.2 -> 85.5 (Simple); 5.3 -> 67.9 (Mid); 0.1 -> 40.5 (Hard) | Helped | Unified Human-Scene Interaction via Prompted Chain-of-Contacts (ICLR 2024 · 4 citations) |
| Add paper-based RAG for unseen tools | Unseen tools · SR | 0.822 -> 0.891 | Helped | ChatHuman: Chatting about 3D Humans with Tools (CVPR 2025 · 3 citations) |
| add appliance manual input | CheckManual Track 1, Total · task planning success rate | 7.99 -> 20.70 | Helped | CheckManual: A New Challenge and Benchmark for Manual-based Appliance Manipulation (CVPR 2025 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 24 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- PartNet-Mobility · 2 of 24 papers
- nuScenes · 2 of 24 papers
Baselines
- GPT-4o · 5 of 24 papers
- LLaVA-1.5 · 2 of 24 papers
- GPT-4o-mini · 2 of 24 papers
- InternVL-3 · 2 of 24 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 52 papers; from OpenAlex and the papers' affiliations).
- University of Science and Technology of China · 4 papers
- Tsinghua University · 4 papers
- Shanghai AI Laboratory · 4 papers
- Massachusetts Institute of Technology · 4 papers
- Peking University · 3 papers
- Stanford University · 3 papers
- Zhejiang University · 3 papers
- Shanghai Artificial Intelligence Laboratory · 3 papers
- The Chinese University of Hong Kong · 3 papers
- Shanghai Jiao Tong University · 3 papers
Use it
Download BibTeX of 25 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Agent instruction-following benchmarks (176 papers)
- Long-horizon LLM agents (155 papers)
- Software engineering agents (140 papers)
- Reasoning benchmarks and platforms (47 papers)
Elsewhere in the topic
- Mathematical and structural reasoning benchmarks (304 papers)
- Inductive and abstract reasoning evaluation (158 papers)
- LLMs for optimization and heuristics (147 papers)
- LLMs for scientific discovery (100 papers)
- Code generation benchmarks (96 papers)
- LLM-guided program synthesis (93 papers)
- Execution-based code verification (90 papers)
- Retrieval-augmented formal theorem proving (62 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).