Research map / Multi-agent systems and games / LLM agents
Agent data synthesis and behavior studies
Synthesizes agent training data and examines behavioral biases in interactive LLM agents.
108 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2024–2026) · an approach in LLM agents, keyword cluster “gui · bench · software”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
8 papers in 2023–24 and 100 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 12.3% to 8.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2024 · FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (NeurIPS 2024 · 37 citations)
- 2025 · BotSim: LLM-Powered Malicious Social Botnet Simulation (AAAI 2025 · 14 citations)
- 2026 · CellAgent: LLM-Driven Multi-Agent Framework for Natural Language-Based Single-Cell Analysis (ICLR 2026 · 42 citations)
Earliest papers here
- FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (NeurIPS 2024 · 37 citations)
- Multi-LLM Debate: Framework, Principals, and Interventions (NeurIPS 2024 · 23 citations)
- AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · 14 citations)
What has been tried
26 papers here report 282 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 4%, hurt 25%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add within-episode CVaR risk control | GOOG · CR % | -1.461 -> 25.077 | Helped | FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (NeurIPS 2024 · 37 citations) |
| add misconception refutation | BoolQ · accuracy | .83±.013 -> .85±.012 | Helped | Multi-LLM Debate: Framework, Principals, and Interventions (NeurIPS 2024 · 23 citations) |
| add planning; parse the video subtitle into subtasks rather than feed the full subtitle to the Actor | AssistGUI overall · success rate | 35.0 -> 46.0 | Helped | AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · 14 citations) |
| neuro-symbolic multi-agent framework with dynamic diagnosis tree (vs direct role-playing gpt-4o) | Human evaluation · Professionalism / Similarity / Fluency (ii) | Professionalism 8.6 vs 6.8; Similarity 8.8 vs 6.4; Fluency (ii) 7.6 vs 5.5 | Helped | MDD-5k: A New Diagnostic Conversation Dataset for Mental Disorders Synthesized via Neuro-Symbolic LLM Agents (AAAI 2025 · 12 citations) |
| Use online rather than offline rule management | ALFWorld · success rate | 90.7 -> 97.4 | Helped | AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning (NeurIPS 2024 · 8 citations) |
| use Universal Block Parsing (UBP) instead of baseline block parsing | Multimodal-Mind2Web · Ele.Acc | Helped | SpiritSight Agent: Advanced GUI Agent with One Look (CVPR 2025 · 3 citations) | |
| Use state abstraction with direct generation and fewer top-ranked elements instead of MCQ with top-50 elements | Mind2Web · success rate | 32% | Helped | Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control (ICLR 2024 · 3 citations) |
| Adding a Judge that verifies new conditions before they enter the condition list (Judge rejects incorrect conditions) | MATH algebra case study · qualitative correctness of condition list | Helped | MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems (NeurIPS 2024 · 3 citations) | |
| Add text control | Film clips test set · FAD | 0.319 -> 0.219 | Helped | FilmComposer: LLM-Driven Music Production for Silent Film Clips (CVPR 2025 · 2 citations) |
| add Feedback to MedChain-Agent without Feedback or MedCase-RAG | MedChain · Average | 0.4341 -> 0.4692 | Helped | MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · 2 citations) |
| use Bandit rather than random action selection | CSI300 · IC | 0.0445 -> 0.0532 | Helped | R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization (NeurIPS 2025 · 1 citation) |
| Add the Perception Agent | Cora · accuracy | 88.5±0.8 -> 93.7±1.0 | Helped | GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environments (NeurIPS 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 100 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- HotpotQA · 3 of 100 papers
- Natural Questions · 2 of 100 papers
- Musique · 2 of 100 papers
- MMLU · 2 of 100 papers
- WebShop · 2 of 100 papers
- MedQA · 2 of 100 papers
- PubMedQA · 2 of 100 papers
- PathVQA · 2 of 100 papers
Baselines
- GPT-4o · 15 of 100 papers
- DeepSeek-R1 · 6 of 100 papers
- GPT-5 · 6 of 100 papers
- Qwen3-32B · 5 of 100 papers
- GPT-4o-mini · 5 of 100 papers
- ReAct · 4 of 100 papers
- Qwen3-8B · 4 of 100 papers
- GPT-4.1 · 4 of 100 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 108 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 4 papers
- Alibaba Group · 4 papers
- Hong Kong University of Science and Technology · 4 papers
- Chinese Academy of Sciences · 3 papers
- University of Chinese Academy of Sciences · 3 papers
- Ant Group · 2 papers
- The University of Hong Kong · 2 papers
- The Chinese University of Hong Kong · 2 papers
- Renmin University of China · 2 papers
- Mohamed bin Zayed University of Artificial Intelligence · 2 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- LLM multi-agent decision making (194 papers)
- Self-evolving language agent learning (188 papers)
- Robust and efficient agent execution (174 papers)
- Agent safety and attack benchmarks (98 papers)
Elsewhere in the topic
- Agentic benchmarks and orchestration (185 papers)
- Software engineering agent benchmarks (129 papers)
- LLM multi-agent collaboration and safety (84 papers)
- GUI agent memory and trajectories (72 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).