Research map / Multi-agent systems and games / LLM agents
LLM multi-agent decision making
Designs collaboration, hierarchy, and credit assignment for LLM-based multi-agent teams.
194 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in LLM agents, keyword cluster “gui · bench · software”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 6 tracked benchmarks.
17 papers in 2023–24 and 176 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 26.2% to 15.2%.
Open questions the data points to
- HumanEval (pass@1): best went from 85.9 (up to 2024) to 92.85 in 2025 — Multi-agent Architecture Search via Agentic Supernet (ICML 2025 · 0 citations)
- MATH (accuracy): best went from 77.5 (up to 2024) to 93 in 2025 — Thought Communication in Multiagent Collaboration (NeurIPS 2025 · 0 citations)
- GSM8K (accuracy) is new: first reported in 2025–26 by 8 papers.
- MMLU (accuracy) is new: first reported in 2025–26 by 6 papers.
- AQuA (accuracy) is new: first reported in 2025–26 by 4 papers.
- MultiArith (accuracy) is new: first reported in 2025–26 by 4 papers.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · Disjunctive Normal Form for Multi-Agent Modal Logics Based on Logical Separability (AAAI 2019 · 4 citations)
- 2024 · MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (ICLR 2024 · 153 citations)
- 2025 · Why Do Multi-Agent LLM Systems Fail? (NeurIPS 2025 · 33 citations)
- 2026 · Multi-Agent Teams Hold Experts Back (ICML 2026 · 2 citations)
Earliest papers here
- Disjunctive Normal Form for Multi-Agent Modal Logics Based on Logical Separability (AAAI 2019 · 4 citations)
- MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (ICLR 2024 · 153 citations)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · 53 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
GSM8K · accuracy (higher is better)
8 papers, 2025–2026 · first reported in 2025–26
| Year | Best so far | Set by |
|---|---|---|
| 2025 | 95.83 | Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems (ICLR 2025 · 3 citations) |
| 2026 | 96.27 | SafeSieve: From Heuristics to Experience in Progressive Pruning for LLM-based Multi-Agent Communication (AAAI 2026 · 0 citations) |
MMLU · accuracy (higher is better)
6 papers, 2025–2026 · first reported in 2025–26
| Year | Best so far | Set by |
|---|---|---|
| 2025 | 84.72 | Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems (ICLR 2025 · 3 citations) |
| 2026 | 92.39 | SafeSieve: From Heuristics to Experience in Progressive Pruning for LLM-based Multi-Agent Communication (AAAI 2026 · 0 citations) |
AQuA · accuracy (higher is better)
4 papers, 2025–2026 · first reported in 2025–26
| Year | Best so far | Set by |
|---|---|---|
| 2025 | 79.47 | G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks (ICML 2025 · 1 citation) |
| 2026 | 91.89 | SafeSieve: From Heuristics to Experience in Progressive Pruning for LLM-based Multi-Agent Communication (AAAI 2026 · 0 citations) |
HumanEval · pass@1 (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 85.9 | MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (ICLR 2024 · 153 citations) |
| 2025 | 92.85 | Multi-agent Architecture Search via Agentic Supernet (ICML 2025 · 0 citations) |
MATH · accuracy (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 77.5 | Offline Training of Language Model Agents with Functions as Learnable Weights (ICML 2024 · 0 citations) |
| 2025 | 93 | Thought Communication in Multiagent Collaboration (NeurIPS 2025 · 0 citations) |
MultiArith · accuracy (higher is better)
4 papers, 2025–2026 · first reported in 2025–26
| Year | Best so far | Set by |
|---|---|---|
| 2025 | 98.8 | Multi-agent Architecture Search via Agentic Supernet (ICML 2025 · 0 citations) |
| 2026 | 98.93 | Assemble Your Crew: Automatic Multi-agent Communication Topology Design via Autoregressive Graph Generation (AAAI 2026 · 2 citations) |
What has been tried
65 papers here report 688 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 5%, hurt 21%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add executable feedback (test-run-debug loop) | HumanEval · Pass@1 | +4.2% | Helped | MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (ICLR 2024 · 153 citations) |
| Add uniqueness loss | Agent-Driver (ChatGPT, contrastive retriever) · ASR-r | 57.4 -> 80.0 | Helped | AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (NeurIPS 2024 · 49 citations) |
| add few-shot examples to the LLM annotator | Held-out human-annotated traces · accuracy | 0.89 -> 0.94 | Helped | Why Do Multi-Agent LLM Systems Fail? (NeurIPS 2025 · 33 citations) |
| Add the planning agent | Ablation evaluation, basic instructions · SR | 59.1 -> 88.6 | Helped | Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration (NeurIPS 2024 · 26 citations) |
| Add link generation and memory evolution | LoCoMo Multi Hop · F1 | 9.65 -> 27.02 | Helped | A-Mem: Agentic Memory for LLM Agents (NeurIPS 2025 · 25 citations) |
| Remove PPO fine-tuning from the shared reflector | HotPotQA, GSM8K, and Checkmate in One Move · success rate | Hurt | Reflective Multi-Agent Collaboration based on Large Language Models (NeurIPS 2024 · 24 citations) | |
| add instruction "you must not trust the other player" | Trust Game · amount sent | -1.26 | Hurt | Can Large Language Model Agents Simulate Human Trust Behavior? (NeurIPS 2024 · 21 citations) |
| Add inference of other parties' preferences | Base game, compromising · Final success (5/6-way) | 60 -> 81 | Helped | Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation (NeurIPS 2024 · 15 citations) |
| Add modeling others | Press Diplomacy vs Cicero (ablation) · Win rate | 0.4% -> 0.7% | Helped | Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy (NeurIPS 2024 · 11 citations) |
| Add CodeActInstruct to fine-tuning data | MINT (ID) · success rate | 17.0 -> 51.3 | Helped | Executable Code Actions Elicit Better LLM Agents (ICML 2024 · 10 citations) |
| Add the RL policy to choose from diverse action candidates | Werewolf, as Villagers · win rate | 0.23 (0.03) -> 0.30 (0.03) | Helped | Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game (ICML 2024 · 8 citations) |
| Increase single-PPO model size from GPT2-Large to GPT2-XL | IMDB Review · KL divergence | Hurt | Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2024 · 8 citations) |
What reviewers will expect
Datasets and baselines named in the 176 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- GSM8K · 34 of 176 papers
- HumanEval · 27 of 176 papers
- MMLU · 26 of 176 papers
- MATH · 25 of 176 papers
- MBPP · 15 of 176 papers
- HotpotQA · 14 of 176 papers
- AQuA · 9 of 176 papers
- MATH-500 · 9 of 176 papers
Baselines
- CoT · 27 of 176 papers
- AFlow · 18 of 176 papers
- Self-Consistency · 17 of 176 papers
- DyLAN · 17 of 176 papers
- LLM-Debate · 15 of 176 papers
- GPTSwarm · 14 of 176 papers
- MetaGPT · 11 of 176 papers
- G-Designer · 11 of 176 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 193 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 13 papers
- University of California, Berkeley · 8 papers
- Peking University · 8 papers
- Nanyang Technological University · 7 papers
- National University of Singapore · 6 papers
- Google DeepMind · 6 papers
- Shanghai Jiao Tong University · 6 papers
- Tongji University · 5 papers
- University of Illinois, Urbana Champaign · 5 papers
- Meta · 5 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Self-evolving language agent learning (188 papers)
- Robust and efficient agent execution (174 papers)
- Agent data synthesis and behavior studies (108 papers)
- Agent safety and attack benchmarks (98 papers)
Elsewhere in the topic
- Agentic benchmarks and orchestration (185 papers)
- Software engineering agent benchmarks (129 papers)
- LLM multi-agent collaboration and safety (84 papers)
- GUI agent memory and trajectories (72 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).