Research map / Multi-agent systems and games / LLM agents

Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents

NeurIPS 2024 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formulates backdoor attacks on ReAct-style LLM agents and instantiates three variants (Query-Attack, Observation-Attack, Thought-Attack) via poisoned fine-tuning data on AgentInstruct WebShop and ToolBench, showing high attack success and that a textual defense (DAN) largely fails.

Problem

LLM-based agents are deployed in high-stakes applications, but their security against backdoor attacks is under-explored; agents' multi-step reasoning opens attack forms beyond the input/output manipulation studied for plain LLMs.

Approach

Generalizes backdoors to intermediate reasoning and action steps of agents: triggers can sit in the user query or in environment observations, and attacks can alter only thoughts/actions while keeping the final output correct (Thought-Attack).

Key results

• Query-Attack-2.1%/10.2% (about 50 poisoned samples at higher relative ratio) reaches 100% ASR on WebShop targets, while clean WS Target ASR is 0%.
• Observation-Attack is harder to activate, reaching 78% ASR at 2.6%/12.5%, but preserves held-in task performance better than Query-Attack.
• The DAN defense yields high false acceptance rates (e.g., 100% FAR for Observation-Attack), indicating existing textual backdoor defenses largely fail on agent backdoors.

Measured effects reported in the paper

Surprising findings

• Observation-Attack achieves lower ASR than Query-Attack, showing triggers hidden in observations are harder for the agent to pick up.
• Attacks on ToolBench Thought-Attack can steer tool choice while preserving translation correctness, so the final output is unchanged.

Limitations

• Experiments cover only one target task per attack form and a single framework (ReAct).
• Backdoored agents sometimes degrade reward on target tasks and affect normal behavior in Query-Attack.
• Trigger sets are narrow; broader triggers yield lower ASR (around 60-68%).

Experimental setup

Datasets: AgentInstruct ToolBench ShareGPT MMLU

Benchmarks: AgentInstruct held-in tasks (AlfWorld, Mind2Web, Knowledge Graph, Operating System, Database) WebShop ToolBench MMLU

Metrics: success rate step success rate F1 reward pass rate attack success rate AUROC false acceptance rate

Baselines: Clean model DAN Mahalanobis distance

Builds on: ReAct AgentInstruct ToolBench DAN

Closest papers by meaning

• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (ICLR 2025 · similarity 0.90)
• AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (NeurIPS 2024 · similarity 0.88)
• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (ICLR 2025 · similarity 0.88)
• Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents (ICLR 2026 · similarity 0.88)
• BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.88)
• UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoning (ICML 2025 · similarity 0.87)