Research map / Multi-agent systems and games / LLM agents

DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DIBench, a paired benign/injected benchmark measuring decision integrity (targeted selection steering, ASR) of GUI mobile agents in multi-candidate selection tasks under non-privileged UI-content injections, and evaluates frameworks, base models, and defenses.

Problem

Existing mobile-agent security benchmarks measure execution-level anomalies (task success, hijack rate) and miss in-task goal deviation, where an agent completes a multi-candidate selection but its final choice is steered by deceptive on-screen content, possibly violating instruction constraints.

Approach

Paired benign/injected instances that change only UI-visible content at the critical selection state, with integrity metrics (TCR, COR, targeted ASR) that separate completion from correctness and attacker-controlled steering, plus a unified configuration suite and defense hooks.

Key results

• Under Combined injection, TCR rises from 42.0% to 72.4% with 45.8% ASR for AppAgent + Qwen2.5-VL on Commerce.
• Mobile-Agent-V3 + Qwen3-VL reaches 52.2% ASR under Combined on Commerce, while closed-source models show ASR as low as 3.8% (GPT-5.2) on Simulator.
• Detection recall drops to 18.2% (Naive) and 21.2% (Escape) versus near-saturated recall on Context Ignore and Combined; prompt-based defenses cut ASR by over 30% in some strong-attack settings but not consistently.

Surprising findings

• Deceptive injections raise task completion and sometimes correctness (COR), so completion-based metrics overestimate trustworthiness.
• Image-processing defenses have inconsistent effects: the same preprocessing can raise ASR in one setting and lower it in another.

Limitations

• Evaluated only on open research mobile-agent frameworks, not commercial or OEM assistants.
• Commercial apps use fixed decision states with screenshot-level injections, not live publishing; single decision point per task.
• Pert. carrier shows low ASR under current implementation, possibly due to rendering disruption.

Experimental setup

Datasets: DIBench benign instances (10 apps x 5 task types x 20)

Benchmarks: DIBench Commerce DIBench Simulator

Metrics: TCR COR ASR detection recall

Baselines: No Attack (benign paired condition) Naive injection Escape injection Context Ignore injection Fake Completion injection

Builds on: AppAgent Mobile-Agent-V2 Mobile-Agent-E Mobile-Agent-V3

Closest papers by meaning

• GhostEI-Bench: Do Mobile Agent Resilience to Environmental Injection in Dynamic On-Device Environments? (ICLR 2026 · similarity 0.89)
• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions (ICML 2026 · similarity 0.87)
• VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.87)
• LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments (NeurIPS 2026 · similarity 0.87)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents (NeurIPS 2026 · similarity 0.87)
• AdapAction: Adaptive Target Action Backdoor Attack against GUI Agents (CVPR 2026 · similarity 0.87)