Research map / Adversarial robustness and security / Attacks on text and graphs
Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection
Wenkui Yang, chao jin, Haisu Zhu, Weilin Luo, Derek Yuen, Kun Shao, Junxian Duan, Huaibo Huang, Jie Cao, Ran He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LLM-guided strategic icon injection instead of random injection | Claude-Sonnet-4.6 victim · GUI grounding 885 split | L1-ASR: 3.23 -> 22.24 | Improved |
| use LLM-guided strategic injection instead of random injection | GUI-Owl-7B victim · GUI grounding 885 split | L1-ASR: 23.36 -> 51.65 | Improved |
| transfer icons optimized on GUI-Owl-7B to UI-TARS-1.5-7B instead of on-target (ablation) | UT/GO-opt. · UI-TARS-1.5-7B | L1-ASR: 32.99 -> 34.43 | No clear effect |
| replace LLM-proposed bounding boxes with random placement (ablation) | Strategic editor · UI-TARS-1.5-7B | L1-ASR: 27.4 -> 33.0 | Improved |
| use iterative carry-forward Depth x Pass@N instead of GPTFuzz-UI / LLM-Bank-Square-UI | Strategic editor · UI-TARS-1.5-7B | L1-ASR: 18.3 -> 33.0 | Improved |
| increase parallel width from pass@1 to pass@3 (ablation) | Strategic editor · GUI-Owl-7B D=1 | L1-ASR: 11.07 -> 24.18 | Improved |
| increase parallel width from pass@1 to pass@3 at depth 5 (ablation) | Strategic editor · GUI-Owl-7B D=5 | L1-ASR: 49.15 -> 50.96 | No clear effect |
| use Claude-Sonnet-4.6 as editor instead of Qwen3-VL-Plus (ablation) | Strategic editor · UI-TARS-1.5-7B | L1-ASR: 33.0 -> 42.4 | Improved |
| use same-family editor as victim (Claude editor vs Claude victim) (ablation) | Strategic editor · Claude-Sonnet-4.6 victim | L1-ASR: 22.2 -> 26.9 | Improved |
| increase model size within Qwen3-VL | Victim · Qwen3-VL | L1-ASR: 38.54 -> 25.47 (2B vs 32B, lower ASR is more robust) | Improved |
| increase model size within specialist OpenCUA | Victim · OpenCUA | L1-ASR: 51.71 -> 57.72 (avg ASR, 7B vs 32B) | Hurt |
Closest papers by meaning
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.88)
- Faithful Mobile GUI Agents with Guided Advantage Estimator (ICML 2026 · similarity 0.88)
- AdapAction: Adaptive Target Action Backdoor Attack against GUI Agents (CVPR 2026 · similarity 0.87)
- GhostEI-Bench: Do Mobile Agent Resilience to Environmental Injection in Dynamic On-Device Environments? (ICLR 2026 · similarity 0.87)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.87)
- GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
