Research map / Adversarial robustness and security / Attacks on text and graphs
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AgentDojo, an extensible environment with 97 user tasks and 629 security test cases for measuring prompt injection attacks and defenses on tool-calling LLM agents.
Problem
Prompt injection attacks hijack LLM agents through untrusted tool outputs, and existing benchmarks either lack multi-step stateful tool calling or rely on LLM simulators that can themselves be fooled by injections.
Approach
A dynamic, stateful agent environment where utility and security are checked by deterministic functions over environment state (not LLM judges), with user tasks crossed with injection tasks and pluggable attacks and defenses.
Key results
- Most capable models solve under 66% of tasks without attack (Claude 3.5 Sonnet 78.22% benign utility is reported in the appendix table).
- GPT-4o targeted ASR with the 'Important message' attack is 47.69% in the main table, and the best adaptive attack reaches roughly 57-58% in ablation.
- Tool filter defense reduces GPT-4o targeted ASR to 6.84%, while the PI detector reduces it to 7.95% at a large utility cost (benign utility 41.49%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Important message phrasing instead of TODO, Ignore previous instructions, or InjecAgent prompts (ablation) | GPT-4o prompt injection attack · AgentDojo security cases | targeted ASR: 3.66% -> 57.7%; 5.41% -> 57.7%; 5.72% -> 57.7% | Helped |
| Select the most effective prompt per task (Max) rather than use one fixed attack prompt (ablation) | GPT-4o prompt injection attack · AgentDojo security cases | targeted ASR: 57.7% -> 57.55% | Helped |
| Include correct user and model names in the injection (ablation) | Important message attack on GPT-4o · AgentDojo security cases | targeted ASR: +1.9% | Helped |
| Address GPT-4o using an incorrect model name (ablation) | Important message attack on GPT-4o · AgentDojo security cases | targeted ASR | Hurt |
| Add data delimiters to tool outputs and instruct the model to ignore enclosed instructions (ablation) | GPT-4o · AgentDojo security cases | targeted ASR: 57. 69% -> 41.65% | Helped |
| Add prompt injection detection and abort when detected (ablation) | GPT-4o · AgentDojo security cases | targeted ASR: 57. 69% -> 7.95% | Helped |
| Add prompt injection detection and abort when detected (ablation) | GPT-4o · AgentDojo utility cases | benign utility: 69.0% -> 41.49% | Hurt |
| Repeat the user prompt after each tool call (ablation) | GPT-4o · AgentDojo security cases | targeted ASR: 57. 69% -> 27.82% | Helped |
| Restrict available tools before observing untrusted data (ablation) | GPT-4o · AgentDojo security cases | targeted ASR: 57. 69% -> 6.84% | Helped |
| Use the Important message attack rather than untargeted denial-of-service attacks (ablation) | GPT-4o agent · AgentDojo security cases | success rate | Helped |
| Place the injection at the end of a tool response (ablation) | GPT-4o prompt injection attack · AgentDojo security cases | targeted ASR: up to 70% | Helped |
Surprising findings
- More capable models tend to be easier to attack (an inverse scaling trend in targeted ASR).
- Many defenses such as data delimiting and prompt repetition increase benign utility, presumably by emphasizing original instructions.
Limitations
- Relies on manually specified tasks and utility checks, which limits scalability.
- Tool filter defense fails when required tools cannot be planned in advance or when task tools suffice for the attack (17% of test cases).
- Attacks and defenses are relatively simple and not adaptive to the defense in most cases.
Experimental setup
Datasets: AgentDojo task suites (Workspace, Slack, Travel, Banking)
Benchmarks: AgentDojo
Metrics: benign utility utility under attack targeted attack success rate untargeted attack success rate
Baselines: Claude 3 Opus Claude 3 Sonnet Claude 3.5 Sonnet GPT-4 Turbo GPT-4o GPT-3.5 Turbo Gemini 1.5 Pro Gemini 1.5 Flash Command R+ Llama 3 70B
Builds on: ToolEmu InjecAgent Berkeley Tool Calling Leaderboard AutoAttack GCG
Closest papers by meaning
- Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents (ICLR 2026 · similarity 0.90)
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (ICLR 2025 · similarity 0.89)
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (ICLR 2025 · similarity 0.89)
- OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.89)
- Memory Injection Attacks on LLM Agents via Query-Only Interaction (NeurIPS 2025 · similarity 0.88)
- DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.