Research map / Multi-agent systems and games / LLM agents
Software engineering agent benchmarks
Builds repository-level environments and benchmarks for coding and computer-use agents.
129 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in LLM agents, keyword cluster “llm agents · llms · agent systems”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
3 papers in 2023–24 and 126 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 4.6% to 10.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · De novo Drug Design using Reinforcement Learning with Multiple GPT Agents (NeurIPS 2023 · 6 citations)
- 2024 · AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls (ICML 2024 · 1 citation)
- 2025 · ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks (ICML 2025 · 4 citations)
- 2026 · U2UData+: A Scalable Swarm UAVs Autonomous Flight Dataset for Embodied Long-horizon Tasks (AAAI 2026 · 7 citations)
Earliest papers here
- De novo Drug Design using Reinforcement Learning with Multiple GPT Agents (NeurIPS 2023 · 6 citations)
- AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls (ICML 2024 · 1 citation)
- R2E: Turning any Github Repository into a Programming Agent Environment (ICML 2024)
What has been tried
28 papers here report 256 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 11%, hurt 17%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Enable reasoning mode in Qwen3-8b | MCPTox · average ASR | average ASR increased by 27.8% (text); 14.0 to 41.8 on the average column | Helped | MCPTox: A Benchmark for Tool Poisoning on Real-World MCP Servers (AAAI 2026 · 7 citations) |
| remove trace data | SRE diagnosis · pass@1 (%) | 13.81% -> 9.52% | Hurt | ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks (ICML 2025 · 4 citations) |
| Add a defensive system prompt | WASP task-related · ASR-intermediate | 0.321 -> 0.167 | Helped | WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks (NeurIPS 2025 · 4 citations) |
| Add self-reflection and episodic memory | RDR2 Follow Micah · successful runs | (1/5) -> (2/5) | Helped | Cradle: Empowering Foundation Agents towards General Computer Control (ICML 2025 · 3 citations) |
| Add an “ignore all restrictions” jailbreak wrapper to deliberate-misuse instructions | Deliberate User Misuse · unsafe execution rate | Hurt | OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents (NeurIPS 2025 · 3 citations) | |
| Add problem statement images | SWE-bench M development split · % Resolved | 8.0 -> 11.0 | Helped | SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? (ICLR 2025 · 3 citations) |
| Enable retrying a different mitigation plan | AIOpsLab mitigation · success rate | 15.4% -> 69.2% | Helped | STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds (NeurIPS 2025 · 2 citations) |
| extending turn budget from 30 to 50 turns | SyncBench time awareness · SR | Independent -0.33%, Collaborative -1.00% | Hurt | SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering (ICML 2025 · 1 citation) |
| Fine-tune on successful trajectories sampled from the policy being updated, mixed with 491 off-policy trajectories | SWE-Bench Lite · resolve rate | 15.3 to 8.7% | Hurt | Training Software Engineering Agents and Verifiers with SWE-Gym (ICML 2025 · 1 citation) |
| add the dual-environment architecture | 420 Python repositories · DGSR | 92.4% -> 100% | Helped | Repo2Run: Automated Building Executable Environment for Code Repository at Scale (NeurIPS 2025 · 1 citation) |
| relax opacity from zero to 0.2, using FI (text) | Full user request leakage, action step · ASR | 0\% to 16\% | Helped | EIA: ENVIRONMENTAL INJECTION ATTACK ON GENERALIST WEB AGENTS FOR PRIVACY LEAKAGE (ICLR 2025 · 1 citation) |
| add a hierarchical API retriever | Filtered ToolBench G2-I and G3-I · pass rate | 22.4 vs 58.9; 15.8 vs 63.2 | Helped | AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls (ICML 2024 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 126 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- HumanEval · 8 of 126 papers
- SWE-bench Verified · 5 of 126 papers
- OSWorld · 4 of 126 papers
- MBPP · 4 of 126 papers
- CodeContests · 3 of 126 papers
- LoCoMo · 3 of 126 papers
- SWE-Gym · 2 of 126 papers
- SWE-bench Multilingual · 2 of 126 papers
Baselines
- GPT-4o · 13 of 126 papers
- OpenHands · 10 of 126 papers
- Claude Code · 9 of 126 papers
- SWE-agent · 8 of 126 papers
- GPT-4o-mini · 7 of 126 papers
- Claude-3.7-Sonnet · 7 of 126 papers
- Claude-3.5-Sonnet · 6 of 126 papers
- Gemini-2.5-Pro · 6 of 126 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 129 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 10 papers
- University of California, Berkeley · 7 papers
- Microsoft · 7 papers
- University of Illinois, Urbana Champaign · 7 papers
- Meta · 7 papers
- Stanford University · 7 papers
- Nanyang Technological University · 5 papers
- Amazon · 5 papers
- Cornell University · 5 papers
- The University of Hong Kong · 4 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Agentic benchmarks and orchestration (185 papers)
- LLM multi-agent collaboration and safety (84 papers)
- GUI agent memory and trajectories (72 papers)
Elsewhere in the topic
- LLM multi-agent decision making (194 papers)
- Self-evolving language agent learning (188 papers)
- Robust and efficient agent execution (174 papers)
- Agent data synthesis and behavior studies (108 papers)
- Agent safety and attack benchmarks (98 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).