Research map / Federated learning and privacy / Differential privacy

GUIGuard-Bench: Toward a General Evaluation for Privacy-Preserving GUI Agents

NeurIPS 2026 · Federated learning and privacy › Differential privacy · approach: Private data analysis frameworks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GUIGuard-Bench, a seed benchmark of 241 real GUI agent trajectories (4,080 screenshots, Android and PC) with region-level privacy, risk, category, and task-necessity labels, and evaluates privacy recognition and planner fidelity under masking.

Problem

GUI agents send screenshots to remote models, exposing sensitive information, but existing visual privacy datasets cover static natural images and GUI benchmarks lack privacy labels and task-necessity judgments.

Approach

A trajectory-level GUI privacy benchmark with region-level bounding boxes, four-level risk labels, six GDPR-based semantic categories, and task-necessary privacy annotations, plus an offline planner-fidelity protocol that replays prerecorded trajectories on protected screenshots and scores consistency with an LLM judge.

Key results

• Binary privacy detection averages 89.0% on Android and 63.3% on PC, but average privacy recall falls to 52.9% (Android) and 13.5% (PC).
• Under strict full-match criteria, average accuracy drops to 8.8% on Android and 0.6% on PC.
• Claude Sonnet 4.6 achieves the highest overall protected planner semantic consistency (2.83 of 4) and 2.98 on Android, while open-source GUI-Owl-7B reaches 1.78 overall.

Surprising findings

• Random square masking yields the highest planner fidelity on average among protection methods, though the paper notes it leaks more information.
• Closed-source general-purpose models outperform GUI-specific agent models under protection, with GUI agent models losing more judgment accuracy when visual information is removed.

Limitations

• Static screenshot evaluation cannot measure privacy protection rates or task completion in dynamic online tasks.
• Planner fidelity is an offline proxy using LLM-as-judge, not end-to-end task success.
• Seed scale; category coverage is unbalanced and the dataset is limited to 241 real trajectories.

Experimental setup

Datasets: GUIGuard-Bench real-environment subset (241 trajectories, 4,080 screenshots) synthetic GUI trajectories (631 total trajectories including synthetic)

Benchmarks: GUIGuard-Bench

Metrics: binary privacy detection accuracy privacy recall risk level accuracy privacy category accuracy task necessity accuracy strict end-to-end accuracy semantic consistency score (0-4) grounding accuracy

Baselines: UI-TARS-1.5-7B Fara-7B GUI-Owl-7B Qwen3.5 GPT-5.3 Gemini 3.1 Pro Claude Sonnet 4.6

Builds on: Agent S3 ScreenSpot GDPR personal data definition

Closest papers by meaning

• MINIM: Privacy-Aware Minimal View for Agents via Trusted Local Sanitization (ICML 2026 · similarity 0.85)
• AdapAction: Adaptive Target Action Backdoor Attack against GUI Agents (CVPR 2026 · similarity 0.84)
• AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents (NeurIPS 2025 · similarity 0.84)
• Contextualized Privacy Defense for LLM Agents (ICML 2026 · similarity 0.83)
• GhostEI-Bench: Do Mobile Agent Resilience to Environmental Injection in Dynamic On-Device Environments? (ICLR 2026 · similarity 0.83)
• DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections (NeurIPS 2026 · similarity 0.83)