Research map / Multi-agent systems and games / LLM agents

GhostEI-Bench: Do Mobile Agent Resilience to Environmental Injection in Dynamic On-Device Environments?

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GhostEI-Bench, a 110-case benchmark that injects overlays, popup SMS and other adversarial UI events into live Android emulator workflows, and uses an LLM judge to measure mobile VLM agents' vulnerability to environmental injection.

Problem

VLM-based mobile GUI agents can be derailed by adversarial UI elements (overlays, spoofed notifications, pop-ups) injected into the environment at runtime, a threat not covered by static or text-prompt safety benchmarks.

Approach

An executable Android emulator benchmark with hook-triggered dynamic injections across seven risk fields, combined with an LLM judge that labels task completion, full/partial attack success and benign failure, and a Vulnerability Rate that excludes benign failures.

Key results

• GPT-5 achieves the lowest Vulnerability Rate at 16.43% with 56.4% task completion, while most other models fall in roughly 40-55% VR.
• Claude-3.7-Sonnet has the highest VR at 55.12%; Gemini-2.5 Pro has 50% TC but 40.00% VR.
• Self-reflection lowers GPT-5-chat-latest VR from 41.67% to 34.62% and GPT-4o VR from 54.87% to 48.75%; GPT-4o judge agrees with human annotation on 93.6% of 110 cases.

Surprising findings

• Explicit thinking mode for Claude-3.7-Sonnet raised VR to 60.0% from 55.12%, while Gemini-2.5 Pro thinking lowered TC without changing VR.
• Mobile-Agent-v2 increased GPT-4o vulnerability relative to AppAgent (54.87% vs 50.00% VR).

Limitations

• Benchmark is small (110 cases) and relies on a single judge model (GPT-4o) validated against one human annotator on one agent's trajectories.
• Judge labels for risky instructions have ambiguous boundaries, as noted in the consistency analysis.
• Only two dynamic attack types (overlay and popup SMS) are evaluated in depth.

Experimental setup

Benchmarks: GhostEI-Bench

Metrics: Task Completion Full Attack Success Partial Attack Success Benign Failure Vulnerability Rate

Baselines: GPT-4o GPT-5 Claude-3.7-Sonnet Claude-Sonnet-4 Gemini-2.5 Pro Qwen2.5-VL-72B-Instruct UI-TARS-7B-SFT UI-TARS-1.5-7B

Builds on: Mobile-Agent-v2 AppAgent OS-HARM

Closest papers by meaning

• DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections (NeurIPS 2026 · similarity 0.89)
• MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control (AAAI 2026 · similarity 0.87)
• AdapAction: Adaptive Target Action Backdoor Attack against GUI Agents (CVPR 2026 · similarity 0.86)
• AndroidReality: How Far Are Mobile Agents from the Real World? (NeurIPS 2026 · similarity 0.86)
• SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.86)
• VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents (ICLR 2026 · similarity 0.86)