Research map / Multi-agent systems and games / LLM agents

AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases

NeurIPS 2024 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 49 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AgentPoison, a backdoor attack that poisons the long-term memory or RAG knowledge base of LLM agents by optimizing a gradient-guided, constrained trigger that maps triggered queries into a unique, compact embedding region, without any model training.

Problem

LLM agents that rely on RAG knowledge bases or memory modules can be compromised by poisoned demonstrations; existing jailbreak and backdoor attacks fail to reliably get malicious demonstrations retrieved while keeping benign performance.

Approach

Cast backdoor trigger generation as constrained optimization: a uniqueness loss pushes triggered query embeddings away from benign key clusters and a compactness loss clusters them, so poisoned demonstrations are retrieved with high probability; target-action and coherence constraints keep the output malicious and the trigger fluent. Optimized with gradient-guided beam search, requiring no model t

Key results

• AgentPoison reaches 81.2% average retrieval attack success rate (ASR-r) across three agents and backbones, versus at most ~56% for baselines in the main table.
• End-to-end target attack success (ASR-t) averages 62.6%, with benign accuracy drop averaging 0.74%, using poison rate below 0.1%.
• A single injected instance with a single-token trigger achieves 62% average ASR-r on one poisoned instance and 79% ASR-r with a one-token trigger.

Measured effects reported in the paper

Surprising findings

• Triggers optimized on one open embedder transfer to other dense retrievers and to the black-box OpenAI text-embedding-ada-002 with high success.
• Perplexity-based and query-rephrasing defenses reduce AgentPoison's attack success only modestly, as its triggers remain inseparable from benign queries in perplexity.

Limitations

• Requires white-box access to the RAG embedder for trigger optimization, though transfer to black-box embedders is shown empirically.
• Evaluated on only three agents and one benchmark each, with small test sets (e.g., 100 to 250 samples).
• Target actions depend on agent-specific safety filters and retrieval requiring all top-k retrieved items to be poisoned.

Experimental setup

Datasets: Agent-Driver dataset StrategyQA EHRAgent memory experiences

Benchmarks: Agent-Driver ReAct-StrategyQA EHRAgent

Metrics: ASR-r ASR-a ASR-t ACC perplexity

Baselines: GCG AutoDAN CPA BadChain

Builds on: GCG AutoDAN BadChain Agent-Driver ReAct

Closest papers by meaning

• UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoning (ICML 2025 · similarity 0.89)
• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (ICLR 2025 · similarity 0.89)
• Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents (NeurIPS 2024 · similarity 0.88)
• Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark (AAAI 2026 · similarity 0.88)
• PROTEUS: A Self-Evolving Red Team with Surface Expansion for Agent Skill Ecosystems (NeurIPS 2026 · similarity 0.88)
• Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents (ICML 2026 · similarity 0.88)