Research map / Multi-agent systems and games / LLM agents
Agentic benchmarks and orchestration
Benchmarks long-horizon agent memory, reliability, and orchestration of sub-agents.
185 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in LLM agents, keyword cluster “llm agents · llms · agent systems”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
6 papers in 2023–24 and 179 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 9.2% to 15.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · Mind2Web: Towards a Generalist Agent for the Web (NeurIPS 2023 · 69 citations)
- 2024 · Secret Collusion among AI Agents: Multi-Agent Deception via Steganography (NeurIPS 2024 · 12 citations)
- 2025 · BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments (ICLR 2025 · 16 citations)
- 2026 · AgentSense: Virtual Sensor Data Generation Using LLM Agents in Simulated Home Environments (AAAI 2026 · 3 citations)
Earliest papers here
- Mind2Web: Towards a Generalist Agent for the Web (NeurIPS 2023 · 69 citations)
- Secret Collusion among AI Agents: Multi-Agent Deception via Steganography (NeurIPS 2024 · 12 citations)
- MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation (ICML 2024 · 9 citations)
What has been tried
35 papers here report 363 measured changes (a component added, removed or swapped, compared with a baseline): helped 73%, no clear effect 6%, hurt 18%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add literature search, AI critic, and gene search | Single-gene experiments, six datasets · hit ratio | +31%, +38%, +10%, +24%, +34%, +4% | Helped | BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments (ICLR 2025 · 16 citations) |
| Use LM-based workflow induction instead of rule-based induction | WebArena · Total SR | 35.6 -> 35.5 | No clear effect | Agent Workflow Memory (ICML 2025 · 10 citations) |
| Fact Check entry in agent response | CIFAR-10 qualitative · hallucinated improvement claims | Helped | MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation (ICML 2024 · 9 citations) | |
| Add privacy-aware system prompt with CoT demonstrations | AgentDAM · privacy performance | 0.638 -> 0.915 | Helped | AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents (NeurIPS 2025 · 5 citations) |
| add Knowledge-Guided Planner | Wood · success rate | 14.29 -> 42.95 | Helped | Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · 5 citations) |
| Replace baseline Cooling Tower control with RL while retaining BG baseline | N=2, m=2, C=5, B=3 · Cooling Tower average power (kW) | 237.31 -> 246.46 | Hurt | LC-Opt: Benchmarking Reinforcement Learning and Agentic AI for End-to-End Liquid Cooling Optimization in Data Centers (NeurIPS 2025 · 4 citations) |
| Add graph to ask | OpenHands judging, DevAI · alignment rate | 65.03% -> 75.95% | Helped | Agent-as-a-Judge: Evaluate Agents with Agents (ICML 2025 · 3 citations) |
| replace the default LLM with Llama 3.1 8B | Multimodal AutoML Agent Benchmark · success rate | 92.0% -> 45.3% | Hurt | MLZero: A Multi-Agent System for End-to-end Machine Learning Automation (NeurIPS 2025 · 3 citations) |
| remove SFT cold start before RL | GAIA · Pass@3 | only 5% | Hurt | WebDancer: Towards Autonomous Information Seeking Agency (NeurIPS 2025 · 3 citations) |
| Use screenshot observations in addition to text (Single-GUI OS vs Single-API OS) | AgentStudio Single-API vs Single-GUI OS tasks · success rate (%) | No clear effect | AgentStudio: A Toolkit for Building General Virtual Agents (ICLR 2025 · 2 citations) | |
| replace ReAct planning with graph-based code planning | HotpotQA · performance | 58.0 -> 64.0 | Helped | MindSearch: Mimicking Human Minds Elicits Deep AI Searcher (ICLR 2025 · 2 citations) |
| Enable the Memory operator | MLE-bench Lite · mean medal rate | No clear effect | AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench (NeurIPS 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 179 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- WebArena · 12 of 179 papers
- SWE-bench Verified · 10 of 179 papers
- Mind2Web · 7 of 179 papers
- HotpotQA · 7 of 179 papers
- ALFWorld · 7 of 179 papers
- GAIA · 6 of 179 papers
- VisualWebArena · 5 of 179 papers
- AndroidWorld · 4 of 179 papers
Baselines
- GPT-4o · 24 of 179 papers
- GPT-5 · 12 of 179 papers
- Gemini 3 Pro · 10 of 179 papers
- Gemini-2.5-Pro · 8 of 179 papers
- Claude-3.7-Sonnet · 8 of 179 papers
- GPT-5.2 · 7 of 179 papers
- Claude Opus 4.5 · 7 of 179 papers
- Mem0 · 7 of 179 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 185 papers; from OpenAlex and the papers' affiliations).
- Meta · 12 papers
- Amazon · 10 papers
- Stanford University · 7 papers
- Microsoft · 7 papers
- Fudan University · 6 papers
- National University of Singapore · 6 papers
- Alibaba Group · 5 papers
- New York University · 5 papers
- Johns Hopkins University · 5 papers
- Google DeepMind · 5 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- LLM multi-agent collaboration and safety (84 papers)
- Software engineering agent benchmarks (129 papers)
- GUI agent memory and trajectories (72 papers)
Elsewhere in the topic
- LLM multi-agent decision making (194 papers)
- Self-evolving language agent learning (188 papers)
- Robust and efficient agent execution (174 papers)
- Agent data synthesis and behavior studies (108 papers)
- Agent safety and attack benchmarks (98 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).