Research map / Adversarial robustness and security / Attacks on text and graphs
MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MELON, a training-free IPI defense that flags an attack when the agent's tool calls under a masked re-execution (user task replaced by a task-neutral prompt) match the original run's tool calls, evaluated on AgentDojo with GPT-4o, o3-mini and Llama-3.3-70B.
Problem
Indirect prompt injection attacks hijack LLM agents through tool-retrieved content, and existing defenses either need model training, fail against sophisticated attacks, or substantially reduce utility.
Approach
Under a successful injection the agent's next tool calls become conditionally independent of the user task; MELON re-executes the trajectory with the user task masked by a task-neutral prompt, caches the masked run's tool calls, and flags an attack when original-run tool calls are embedding-similar to cached ones.
Key results
- On AgentDojo with GPT-4o, MELON-Aug reduces average ASR to 0.32% while keeping 68.72% utility under attack (no defense: 16.06% ASR, 69.08% UA).
- MELON achieves 0.24% average ASR on GPT-4o with 58.78% UA; the LLM detector reaches 0% ASR but also 0% UA.
- MELON's false positive rate on GPT-4o is 9.28%, versus 61.86% for the DeBERTa detector and 0.00% for the LLM detector.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Repeat Prompt augmentation to MELON | MELON on GPT-4o · AgentDojo, average | UA: 58.78% -> 68.72% | Helped |
| add Repeat Prompt augmentation to MELON | MELON on GPT-4o · AgentDojo, average | ASR: 0.24% -> 0.32% | Hurt |
| replace the user task with a task-neutral prompt in the masking function (ablation) | MELON · AgentDojo, Important Messages attack, GPT-4o | ASR: 2.70% -> 0.95% | Helped |
| add a cache of masked-run tool calls (ablation) | MELON · AgentDojo, Important Messages attack, GPT-4o | ASR: 1.75% -> 0.95% | Helped |
| compare only tool calls rather than complete actions (ablation) | MELON · AgentDojo, Important Messages attack, GPT-4o | ASR: 17.33% -> 0.95% | Helped |
| use a paraphrased task-neutral prompt (ablation) | MELON · AgentDojo, Important Messages attack, GPT-4o | ASR: 0.95% -> 1.11% | Hurt |
| use a sentiment-analysis task-neutral prompt (ablation) | MELON · AgentDojo, Important Messages attack, GPT-4o | ASR: 0.95% -> 1.11% | Hurt |
| use a grammar-checking task-neutral prompt (ablation) | MELON · AgentDojo, Important Messages attack, GPT-4o | ASR: 0.95% -> 0.95% | No clear effect |
| use a translation task-neutral prompt (ablation) | MELON · AgentDojo, Important Messages attack, GPT-4o | ASR: 0.95% -> 1.43% | Hurt |
| set the similarity threshold to 0.5 instead of 0.8 (ablation) | MELON · AgentDojo, Important Messages attack, GPT-4o | UA: 32.91% -> 30.21% | Hurt |
| set the similarity threshold to 0.6 instead of 0.8 (ablation) | MELON · AgentDojo, Important Messages attack, GPT-4o | UA: 32.91% -> 31.32% | Hurt |
| set the similarity threshold to 0.7 instead of 0.8 (ablation) | MELON · AgentDojo, Important Messages attack, GPT-4o | ASR: 0.95% -> 0.95% | No clear effect |
Surprising findings
- Tool Filter and LLM Detector reach 0% ASR only by blocking nearly all tool use, yielding near-zero utility under attack.
- Most MELON failure cases (72.73% of evading attacks) are response-based attacks that manifest in text rather than tool calls.
Limitations
- Misses response-based attacks since only tool calls are compared (72.73% of evading cases).
- Non-trivial false positives on user tasks that explicitly ask to execute external instructions, and about 2x API cost.
Experimental setup
Datasets: AgentDojo
Benchmarks: AgentDojo
Metrics: Benign Utility Utility under Attack Attack Success Rate false positive rate
Baselines: No Defense Delimiting Repeat Prompt Tool Filter DeBERTa Detector LLM Detector
Builds on: AgentDojo Repeat Prompt defense Tool Filter
Closest papers by meaning
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (NeurIPS 2024 · similarity 0.86)
- CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attribution (ICML 2026 · similarity 0.85)
- Learning to Inject: Automated Prompt Injection via Reinforcement Learning (NeurIPS 2026 · similarity 0.85)
- MCPTox: A Benchmark for Tool Poisoning on Real-World MCP Servers (AAAI 2026 · similarity 0.85)
- Think Twice Before You Act: Protecting LLM Agents Against Tool Description Poisoning via Isolated Planning (ICML 2026 · similarity 0.85)
- VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.