Research map / Federated learning and privacy / Privacy attacks and secure learning
Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agents
Zhixin Lin, Jungang Li, Shidong Pan, Yibo Shi, Yue Yao, Dongliang Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add implicit hint to prompt (ablation) | Qwen2.5-VL 7B · SAPA-Bench | RA: 11.75 -> 22.67 | Improved |
| add implicit hint to prompt (ablation) | InternVL 2.5 8B · SAPA-Bench | RA: 14.88 -> 28.70 | Improved |
| add explicit hint instead of implicit (ablation) | GPT-4o · SAPA-Bench | RA: 29.40 -> 55.03 | Improved |
| add explicit hint instead of implicit (ablation) | Gemini 2.0-flash · SAPA-Bench | RA: 27.37 -> 67.14 | Improved |
| higher scenario sensitivity level | GPT-4o · SAPA-Bench | PRR: 67.66 -> 89.19 (Low -> High) | Improved |
| higher scenario sensitivity level | Gemini 2.0-flash · SAPA-Bench | PRR: 55.94 -> 90.81 (Low -> High) | Improved |
| higher scenario sensitivity level | Show-UI · SAPA-Bench | PRR: 31.52 -> 36.42 (Low -> High) | Improved |
| higher scenario sensitivity level | SpiritSight Agent · SAPA-Bench | PRR: 34.16 -> 31.05 (Low -> High) | Hurt |
| closed-source vs open-source backbone | Gemini 2.0-flash / GPT-4o · SAPA-Bench | RA: 67.14 and 55.03 vs best open 51.66 | Mixed |
| higher privacy awareness vs task success | SpiritSight Agent vs Gemini 2.0-flash · GUI-Odyssey | SR: SR 43.00 vs 48.12 | No clear effect |
Closest papers by meaning
- AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents (NeurIPS 2025 · similarity 0.86)
- Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmark (ICLR 2026 · similarity 0.86)
- PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action (NeurIPS 2024 · similarity 0.86)
- EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automation (AAAI 2026 · similarity 0.85)
- MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models (ICML 2026 · similarity 0.85)
- SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
