Research map / Multi-agent systems and games / LLM agents

D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes D-GARA, a dynamic Android benchmarking framework that injects real-world interruptions (permission dialogs, low battery, update prompts, crashes) into live GUI-agent task execution, and releases the 152-task D-GARA-152 benchmark showing large success-rate drops for SOTA agents.

Problem

Existing GUI agent benchmarks are static or idealized and do not evaluate robustness to unexpected real-world interruptions and anomalies that can redirect execution during live Android interaction.

Approach

A rule-based semantic anomaly trigger injects configurable interruptions (via a dedicated APK and ADB) into live Android task execution when XML keywords match, with follow-up state transitions, and validates task success from the final UI state via declarative XML goal rules rather than the agent's "done" signal.

Key results

• Gemini2.5-flash SR drops from 80.26% without interruptions to 68.42% with interruptions (RSR 73.77%).
• Qwen2.5-VL-7B SR falls from 69.08% to 46.05% (RSR 53.33%); AgentCPM-GUI-8B falls from 59.87% to 26.97% (RSR 39.56%).
• Qwen2.5-VL-7B RSR is 96.15% on dual-button interruptions but 41.27% on single-button ones; AgentCPM-GUI-8B drops to 9.30% on single-button.

Surprising findings

• Agents mostly bypass interruptions by choosing 'Close', and performance drops sharply when forced to take the complex path (e.g., 'Install Now').
• Screenshot-only input causes Gemini2.5-flash to drop ~35 points in success rate, indicating coordinate inference from vision is a bottleneck.

Limitations

• Benchmark is small (152 tasks across 8 apps) and Android-only.
• Success validation relies on hand-crafted XML rules with human review of automatic results.
• Anomaly triggers are keyword-rule based rather than fully dynamic or learned.

Experimental setup

Datasets: D-GARA-152

Benchmarks: D-GARA-152

Metrics: Success Rate (SR) Robust Success Rate (RSR)

Baselines: UI-TARS-1.5-72B AgentCPM-GUI-8B GPT-4o Gemini2.5-flash Qwen2.5-VL-7B

Builds on: AndroidWorld GUI-Robust ADB

Closest papers by meaning

• Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · similarity 0.89)
• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.89)
• VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.89)
• ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.88)
• AdapAction: Adaptive Target Action Backdoor Attack against GUI Agents (CVPR 2026 · similarity 0.87)
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (ICLR 2025 · similarity 0.87)