Research map / Multi-agent systems and games / LLM agents

Contextualized Privacy Defense for LLM Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Contextualized Defense Instructing (CDI), where a lightweight instructor model gives step-specific privacy guidance to a tool-using agent, trained with GRPO on failure trajectories from adversarial attacks, and compares it with prompting and guarding in a simulated email/messaging privacy benchmark.

Problem

LLM agents acting on users' personal information in multi-step tool use leak sensitive data to adaptive attackers, and existing static (prompting) or passive (guarding) defenses cannot give contextual, proactive privacy guidance or withstand strategic attacks.

Approach

An instructor model inserts context-aware privacy guidance after each tool result to steer the agent's next action, and is optimized with GRPO by converting attack-induced privacy-leak trajectories into truncated RL environments, using a warm-up PP reward followed by an appropriate-disclosure (AD) reward.

Key results

• Optimized CDI reaches PP 94.2%, HS 80.6%, AD 86.5% on 100 unseen test configurations, versus 36.8% PP for no defense.
• Under adversarial attack, optimized CDI keeps PP 79.5% while optimized prompting drops from 89.0% to 55.1% and optimized guarding from 80.2% to 50.3%.
• Without optimization, CDI raises PP from 35.5% to 75.9% (table reports 70.0% PP on test for unoptimized CDI in the averaged table), with the best unoptimized AD of 82.8% before attack.

Surprising findings

• Safety-aligned guard models (Qwen3-4B-SafeRL, gpt-oss-safeguard-20B) do not markedly improve privacy over their base versions.
• Training CDI with PP alone sharply reduces helpfulness, and direct AD training causes a cold-start problem requiring a PP warm-up.

Limitations

• Evaluation is limited to a single simulated communication-app environment with synthetic configurations generated by gpt-5.
• Privacy norms are assumed ground truth and labeled with LLM judges, with the instructor not given the norms at test time.
• Results depend on a single agent backbone (gpt-4.1-mini) for most experiments, and the attack algorithm was optimized on CDI with Qwen3-4B.

Experimental setup

Datasets: PrivacyLens 115 simulation configurations (100 test, 15 for defense optimization)

Benchmarks: in-house privacy risk simulation (100 test configurations)

Metrics: privacy preservation rate (PP) helpfulness score (HS) appropriate disclosure (AD)

Baselines: No defense (N/A) Prompting Guarding Qwen3Guard-4B-SafeRL gpt-oss-safeguard-20B

Builds on: ReAct GRPO PrivacyLens TRL

Closest papers by meaning

• Contextual Integrity in LLMs via Reasoning and Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training (ICML 2026 · similarity 0.90)
• Searching for Privacy Risks in LLM Agents via Simulation (ICLR 2026 · similarity 0.90)
• It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs (NeurIPS 2026 · similarity 0.89)
• MINIM: Privacy-Aware Minimal View for Agents via Trusted Local Sanitization (ICML 2026 · similarity 0.88)
• AI Agents May Always Fall for Prompt Injections (NeurIPS 2026 · similarity 0.88)