Research map / Multi-agent systems and games / LLM agents

POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces POLAR-Bench, a 7,852-sample benchmark scoring privacy and utility deterministically for trusted LLM agents facing adversarial third-party probing, varying privacy policy type and attack strategy on a 5x5 grid.

Problem

Trusted LLM agents holding private user data must share task-relevant attributes while withholding protected ones when third-party systems adversarially probe for them; existing benchmarks do not jointly vary policy type and attack strategy or score privacy and utility together.

Approach

A two-agent benchmark (trusted model vs. adversarially prompted third-party model) with typed attributes, graded privacy policy dimensions (P1-P5) and graded attack strategies (S1-S5) as orthogonal axes, scored by deterministic regex set-membership rather than an LLM judge.

Key results

• GLM-5.1 achieves the top overall score of 94.6, ahead of GPT-5.4 (92.6) and Gemma-4-31B (91.4).
• Frontier models withhold over 99% of protected attributes while the weakest open-weight models leak over half.
• GPQA Diamond correlates with privacy (Pearson r=0.752) but only weakly and negatively with utility (r=-0.291) across 20 models.
• PrivacyChecker raises Ministral-3-3B privacy score from 48.27 to 75.79 at a cost of 2.72 utility points.

Surprising findings

• Model size is not a reliable predictor: Gemma-4-E4B outscores GPT-OSS-120B and Apertus-70B on overall score, and DeepSeek-R1-Distill-Llama-70B leaks more than much smaller models.
• Multi-turn progressive attacks keep utility highest yet still substantially reduce privacy, while prompt-injection attacks behave more consistently across models.

Limitations

• Samples are generated from structured profiles rather than real user logs, and the policy and attack difficulty ordering is not validated with human reviewers.
• All attacker prompts are generated by a single Llama-3.3-70B model, so attack strategies may shift under other attackers.
• Deterministic regex scoring is blind to paraphrased disclosures, especially under abstraction and conflict policies.

Experimental setup

Datasets: POLAR-Bench synthetic samples

Benchmarks: POLAR-Bench GPQA Diamond

Metrics: privacy score utility score overall score Pearson r Spearman rho

Baselines: PrivacyChecker AirGapAgent AgentLeak PrivacyLens ConfAIde AGENTDAM PAPILLON

Builds on: AirGapAgent AgentLeak PrivacyChecker AGENTDAM

Closest papers by meaning

• MINIM: Privacy-Aware Minimal View for Agents via Trusted Local Sanitization (ICML 2026 · similarity 0.88)
• Contextualized Privacy Defense for LLM Agents (ICML 2026 · similarity 0.87)
• A Unified Framework for Adversary-Aware Differential Privacy Bounds (NeurIPS 2026 · similarity 0.87)
• Searching for Privacy Risks in LLM Agents via Simulation (ICLR 2026 · similarity 0.87)
• Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents (ICLR 2026 · similarity 0.86)
• Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.86)