Research map / Multi-agent systems and games / LLM agents

AI Agents May Always Fall for Prompt Injections

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Argues prompt injection should be framed as a Contextual Integrity violation, showing current classifiers, alignment training, and data-instruction separation fail on context-manipulation attacks, and that an impossibility tradeoff follows.

Problem

Prompt injection defenses rely on data-instruction separation and injection classifiers, which fail to detect attacks that manipulate context and degrade contextually appropriate agent behavior.

Approach

Recasts prompt injection as a Contextual Integrity violation: an agent action is an information flow whose appropriateness depends on five parameters (sender, receiver, subject, information type, transmission principle) and on norm legitimacy, so attacks supply false context values; argues an impossibility result that fixed allow/block rules trade off legitimate flows against attacks.

Key results

• Iterative CI-grounded red-teaming reaches 96.7% attack success against the email assistant, vs. under 1% for the static baseline emails.
• Prompt Guard and Prompt Guard 2 achieve near-chance AUROC of 0.43-0.59 on paired authority-asserting vs. neutral emails.
• Without explicit boundaries, gpt-5.2 crosses the F2 boundary in 65% of simultaneous-flow scenarios, versus 6% for claude-sonnet-4-6.

Surprising findings

• SecAlign-70B sends emails more often than the LLaMA baseline under restrictive delegation (88.2% vs 54.0% on authority-asserting emails), worsening contextual appropriateness.
• SecAlign flags only 54.6% of injections in LLMail-Inject versus 89.8% for the LLaMA baseline, indicating safety training reduces engagement with external content.

Limitations

• Impossibility result is informal rather than formally proven.
• Norm-evaluation attacks are only qualitatively analyzed because the automated attacker does not reliably respect its constraints.
• Experiments assume explicit delegation statements and ground-truth labels for appropriateness, which real deployments may not satisfy.

Experimental setup

Datasets: LLMail-Inject ConVerse

Benchmarks: LLMail-Inject

Metrics: AUROC send rate attack success rate execution rate utility rate security violation rate

Baselines: Prompt Guard Prompt Guard 2 LLaMA-3.3-70B-Instruct SecAlign-70B ConVerse

Builds on: PAIR Contextual Integrity AgentDojo LLMail-Inject SecAlign

Closest papers by meaning

• Optimizing Agent Planning for Security and Autonomy (ICLR 2026 · similarity 0.89)
• Contextual Integrity in LLMs via Reasoning and Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attribution (ICML 2026 · similarity 0.88)
• Contextualized Privacy Defense for LLM Agents (ICML 2026 · similarity 0.88)
• ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents (NeurIPS 2026 · similarity 0.87)
• It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs (NeurIPS 2026 · similarity 0.86)