Research map / Multi-agent systems and games / LLM agents
Agent safety and attack benchmarks
Benchmarks multi-turn attacks, prompt injection, and malicious tool use against LLM agents.
98 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in LLM agents, keyword cluster “gui · bench · software”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
0 papers in 2023–24 and 95 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 0.0% to 8.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Incentivising Monitoring in Open Normative Systems (AAAI 2017 · 5 citations)
- 2020 · COBRA: Context-Aware Bernoulli Neural Networks for Reputation Assessment (AAAI 2020 · 7 citations)
- 2021 · Verifiable Machine Ethics in Changing Contexts (AAAI 2021 · 14 citations)
- 2025 · Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systems (AAAI 2025 · 8 citations)
- 2026 · Securing AI Agents with Information-Flow Control (NeurIPS 2026 · 3 citations)
Earliest papers here
- Incentivising Monitoring in Open Normative Systems (AAAI 2017 · 5 citations)
- COBRA: Context-Aware Bernoulli Neural Networks for Reputation Assessment (AAAI 2020 · 7 citations)
- Verifiable Machine Ethics in Changing Contexts (AAAI 2021 · 14 citations)
What has been tried
11 papers here report 114 measured changes (a component added, removed or swapped, compared with a baseline): helped 54%, no clear effect 12%, hurt 28%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use temperature 1 sampling and select the best of 5 attempts | AgentHarm public test · harm score | 70.7%±0.7% -> 83.7% | Helped | AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (ICLR 2025 · 5 citations) |
| Add prior poisoning to the memory bank before the subsequent injection | MIMIC-III (Pair 2) · ISR | 100.0 -> 93.3 | Hurt | Memory Injection Attacks on LLM Agents via Query-Only Interaction (NeurIPS 2025 · 3 citations) |
| Vary declared tool parameter count from 0 to 10 | Tool-use scenarios · ASR | No clear effect | Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Tools (NeurIPS 2025 · 1 citation) | |
| Add EAgent attack-effectiveness supervision on top of naive GAgent | nuImage Env 1 (threshold 0.5) · ASR avg | 5.33% -> 45.00% | Helped | MAGIC: Mastering Physical Adversarial Generation in Context Through Collaborative LLM Agents (AAAI 2026 · 0 citations) |
| Add predictive ranking to SelfRef | VALID · EESR | 55.6 -> 60.8 | Helped | TAI3: Testing Agent Integrity in Interpreting User Intent (NeurIPS 2025 · 0 citations) |
| Increase repeated samples k from 1 to 10 | InterCode CTF (Test), NYU CTF (Test), and Cybench · pass@k | Helped | Dynamic Risk Assessments for Offensive Cybersecurity Agents (NeurIPS 2025 · 0 citations) | |
| remove toolbox and memory | EICU-AC · LPA | 98.7 -> 90.8 | Hurt | GuardAgent: Safeguard LLM Agents via Knowledge-Enabled Reasoning (ICML 2025) |
| Remove explicit reasoning prompts | Who&When, algorithm-generated systems · Agent-Level Accuracy | -7.4% | Hurt | Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems (ICML 2025) |
What reviewers will expect
Datasets and baselines named in the 95 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- AgentDojo · 12 of 95 papers
- MMLU · 5 of 95 papers
- StrongREJECT · 4 of 95 papers
- WebShop · 4 of 95 papers
- OSWorld · 4 of 95 papers
- eICU · 3 of 95 papers
- SHADE-Arena · 3 of 95 papers
- Agent-SafetyBench · 3 of 95 papers
Baselines
- No defense · 4 of 95 papers
- GPT-4o · 4 of 95 papers
- Claude Sonnet 4.5 · 3 of 95 papers
- DRIFT · 3 of 95 papers
- LlamaFirewall · 3 of 95 papers
- DeepSeek-V3 · 3 of 95 papers
- GPT-5.2 · 3 of 95 papers
- Gemini 2.5 Pro · 3 of 95 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 95 papers; from OpenAlex and the papers' affiliations).
- Nanyang Technological University · 5 papers
- University of Oxford · 4 papers
- University of Illinois, Urbana Champaign · 3 papers
- University of Georgia · 3 papers
- University of California, Berkeley · 3 papers
- Microsoft · 3 papers
- National University of Singapore · 3 papers
- Harvard University · 2 papers
- Amazon · 2 papers
- Stanford University · 2 papers
Use it
Download BibTeX of 23 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- LLM multi-agent decision making (194 papers)
- Self-evolving language agent learning (188 papers)
- Robust and efficient agent execution (174 papers)
- Agent data synthesis and behavior studies (108 papers)
Elsewhere in the topic
- Agentic benchmarks and orchestration (185 papers)
- Software engineering agent benchmarks (129 papers)
- LLM multi-agent collaboration and safety (84 papers)
- GUI agent memory and trajectories (72 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).