Research map / Multi-agent systems and games / LLM agents

Searching for Privacy Risks in LLM Agents via Simulation

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a search-based framework that alternates LLM-optimized attack and defense instructions in multi-agent privacy simulations, discovering escalating attacks (impersonation, consent forgery) and defenses (identity-verification state machines).

Problem

LLM agents may be targeted by malicious agents that engage them in multi-turn dialogues to extract sensitive information, and such evolving agent-agent privacy risks are hard to anticipate or defend against manually.

Approach

Treat attacker and defender agent instructions as optimizable objects in a simulated privacy environment; use an LLM optimizer to reflect on trajectories, with parallel search threads and cross-thread propagation for attacks, and single-thread search for universal defenses, alternating between the two.

Key results

• Alternating search raises attack leak velocity from 3.4% (A0 vs D0) to 76.0% (A1 vs D0), and defenses cut it to 2.5% (A1 vs D1) before impersonation attack A2 reaches 42.2% against D1.
• Final defense D2 holds A2 to 7.1% leak velocity on Training-5 and 2.9% on Testing-100 with in-context transfer.
• Defenses discovered with gpt-4.1-mini transfer across backbones; gpt-4.1 as defender lowers basic-attack leak velocity to 16.5% vs 31.2% for gpt-4.1-mini.

Surprising findings

• Sending a naive impersonation message from the recipient's own account is highly effective against LLM agents, though it would not work on humans.
• Stronger backbones give only slight attack gains but substantially better defense; longer thinking does not reliably improve privacy awareness.

Limitations

• Simulation uses mock applications and synthetic personas; real-world transfer shown only in a small case study (3/5 trials in ChatGPT Atlas).
• Defenses are prompt-based only; more complex defense protocols are left to future work.
• Data subject disabled after transfer, so no consent is ever granted in simulation.

Experimental setup

Datasets: PrivacyLens-style privacy norms Testing-100 Training-5

Benchmarks: Testing-100

Metrics: leak velocity leak rate human agreement

Baselines: basic instructions (A0, D0) comprehensive defense without search sequential search (N=1)

Builds on: ReAct PrivacyLens contextual integrity reflection-based prompt optimization evolutionary migration

Closest papers by meaning

• Contextualized Privacy Defense for LLM Agents (ICML 2026 · similarity 0.90)
• From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents (ICML 2026 · similarity 0.87)
• ORBIT: A Framework for Multi-Agent Security Evaluations (NeurIPS 2026 · similarity 0.87)
• POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents (NeurIPS 2026 · similarity 0.87)
• Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents (ICLR 2026 · similarity 0.86)
• SafeHarbor: Defining Precise Decision Boundaries via Hierarchical Memory-Augmented Guardrail for LLM Agent Safety (ICML 2026 · similarity 0.86)