Research map / Multi-agent systems and games / LLM agents
Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ran a one-month public red-teaming competition (Gray Swan Arena, March-April 2025) against 22 frontier LLM agents across 44 deployment scenarios, collecting 1.8M prompt injections and building the ART benchmark of 4,700 attacks evaluated on 19 models.
Problem
Whether AI agents powered by frontier LLMs can keep to deployment policies when attacked with prompt injections in realistic tool-using environments, and how robustness relates to model size, capability, and inference compute.
Approach
Large-scale, realistic public red-teaming of tool-using agents with policy-based success criteria, distilled into a curated, transferable attack benchmark (ART) with an analysis of attack transfer, universality, and robustness factors.
Key results
- Competition collected about 1.8M prompt-injection attempts and over 62,000 successful policy violations across 22 frontier LLMs; overall ASR 12.7%, indirect injections 27.1% vs direct 5.7%.
- ART benchmark contains 4,700 selected attacks across 44 behaviors; nearly all agents violate policies within 10-100 queries per behavior.
- Robustness shows limited correlation with model capability, release date, or inference compute (e.g., o3-mini vs o4-mini, GPT-4o vs GPT-4.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use indirect rather than direct prompt injection | AI agents in the red-teaming challenge · All categories | attack success rate: 5.7 -> 27.1 | Helped |
| Use indirect rather than direct prompt injection | AI agents in the red-teaming challenge · Confidentiality Breaches | attack success rate: 7.8 -> 29.8 | Helped |
| Use indirect rather than direct prompt injection | AI agents in the red-teaming challenge · Prohibited Content | attack success rate: 4.8 -> 15.9 | Helped |
| Use indirect rather than direct prompt injection | AI agents in the red-teaming challenge · Prohibited Action | attack success rate: 4.2 -> 36.8 | Helped |
| Enable reasoning capabilities | Claude 3.7 Sonnet · AI agent red-teaming evaluation | robustness | Helped |
| Use extended reasoning time | o3-mini · AI agent red-teaming evaluation | robustness | No clear effect |
| Use reasoning-capable Gemini variants | Gemini models · AI agent red-teaming evaluation | robustness | No clear effect |
| Increase model capability or size | frontier AI agents · AI agent red-teaming evaluation | robustness | No clear effect |
Surprising findings
- Robustness shows little correlation with model size, capability, or inference-time compute.
- Attacks that succeed on more robust models tend to transfer well to less robust ones.
Limitations
- Results depend on simulated tool environments and anonymized agents rather than fully live deployments.
- Judging relied on automated LLM and programmatic judges with human appeals, which may introduce error.
- ART is kept as a private leaderboard, with only partial public release planned.
Experimental setup
Datasets: ART red-teaming competition submissions (1.8M attacks)
Benchmarks: ART benchmark
Metrics: attack success rate behavior ASR transfer ASR
Baselines: AgentDojo AgentHarm InjecAgent
Builds on: AgentDojo AgentHarm InjecAgent
Closest papers by meaning
- Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming (NeurIPS 2025 · similarity 0.88)
- AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration (NeurIPS 2025 · similarity 0.87)
- AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (NeurIPS 2024 · similarity 0.87)
- PROTEUS: A Self-Evolving Red Team with Surface Expansion for Agent Skill Ecosystems (NeurIPS 2026 · similarity 0.87)
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (NeurIPS 2024 · similarity 0.86)
- Automated Red Teaming with GOAT: the Generative Offensive Agent Tester (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.