Research map / Multi-agent systems and games / LLM agents

Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmark

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EAPrivacy, a four-tier procedurally generated PDDL/multimodal benchmark measuring physical-world privacy awareness of LLM-powered embodied agents, finding large deficits across 16 models.

Problem

Existing LLM privacy benchmarks only cover natural-language scenarios, leaving privacy awareness of LLM-powered embodied agents in physical environments (object sensitivity, changing contexts, inferred privacy under task conflict, social norm vs. privacy) unmeasured.

Approach

A procedurally generated, four-tier benchmark in which scenes are given to LLMs as structured PDDL states plus simulated multimodal cues (visual/audio), testing sensitive object identification, contextual appropriateness, inferential privacy under task conflict, and social norm vs. privacy dilemmas.

Key results

• Gemini 2.5 Pro achieved the best Tier 2 selection accuracy at 59%, and models prefer neutral over most-optimal actions.
• In Tier 3, privacy violation rates range from 71% to 98%, with most models above 70% and near-zero task completeness for most models.
• In Tier 4, gpt-5-low and gpt-5-high reach 100% selection accuracy, while several models disregard the social norm over 15% of the time.

Surprising findings

• Thinking-enabled variants (Gemini, Qwen) often underperform their non-thinking counterparts across tiers.
• Models perform substantially worse in PDDL physical settings than in equivalent text-only settings on Tier 1.

Limitations

• Simulated environments rather than real robots
• Small annotator pool of five US-based PhD-level raters; US-centric norms
• Human agreement and ratings on a limited set of items

Experimental setup

Datasets: EAPrivacy procedurally generated scenarios sensitive information catalog from prior privacy literature

Benchmarks: EAPrivacy

Metrics: Main Object Ratio Objects Not On Container Mean Absolute Difference Selection Accuracy Privacy Violation Rate Task Completeness Rating Accuracy

Baselines: claude-3.5-haiku gemini-2.5-flash gemini-2.5-pro gpt-4o-mini gpt-4o gpt-5 gpt-oss-120b Llama-3.3-70B

Builds on: contextual integrity PDDL

Closest papers by meaning

• PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action (NeurIPS 2024 · similarity 0.89)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.89)
• Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.87)
• Privacy Reasoning in Ambiguous Contexts (NeurIPS 2025 · similarity 0.87)
• Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-Based Decision-Making Systems (ICLR 2025 · similarity 0.87)
• MINIM: Privacy-Aware Minimal View for Agents via Trusted Local Sanitization (ICML 2026 · similarity 0.87)