Research map / Multi-agent systems and games / LLM agents

Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GUI-RobustEval, a 1,216-case benchmark for recovering from policy-induced GUI agent errors, and RoTS, a tree-based trajectory synthesis framework producing 800k SFT samples that fine-tune Qwen2.5-VL-7B/32B to state-of-the-art OSWorld results among open-weights models.

Problem

GUI agents lack robustness to their own policy-induced errors, and existing benchmarks and training data under-cover realistic error types and long-horizon error depths.

Approach

Build a replayable trajectory tree from online rollouts and expand it with fragility-driven exploration on successful branches and experience-informed recovery on failed branches, yielding diverse long-horizon failure-recovery training data; pair this with a benchmark that injects realistic erroneous prefixes at controlled depths.

Key results

• RoTS-32B reaches 47.4% success on OSWorld at max 50 steps and 33.8% All-Pass@4, state-of-the-art among open-weights models in the paper's comparison.
• On GUI-RobustEval at error depth 5, RoTS-32B achieves 33.2% success vs 25.9% for OpenCUA-32B, with the smallest relative drop (33%).
• Ablation: FDE+EIR co-expansion raises OSWorld success from 18.1 to 21.4 and All-Pass@4 from 8.6 to 14.1 at a fixed 100k budget.

Surprising findings

• Adding human-curated reflection data (AgentNet reflection subset) gives only small gains, while policy-induced reflection data gives much larger gains.
• Increasing reflection data ratio beyond 0.1 hurts performance (14.8% at 0.3, below no-reflection baseline), suggesting over-reflection wastes budget.

Limitations

• Focuses only on desktop computer-use tasks; mobile and edge devices not evaluated.
• Cross-format conversion of injected prefixes for heterogeneous agents may confound within-agent depth trends.
• Dataset performance saturates at about 1000k samples under the current tree expansion setting.

Experimental setup

Datasets: AgentNet AgentTrek GUI-Reflection OSWorld tasks (synthesis)

Benchmarks: GUI-RobustEval OSWorld-Verified WindowsAgentArena

Metrics: error-awareness rate post-error success rate success rate All-Pass@4

Baselines: GPT 5.1 Jedi-7B Qwen3-VL-Plus Qwen3-VL-Flash Qwen3-VL-8B-Instruct UI-TARS-1.5-7B OpenCUA-7B OpenCUA-32B GUI-Owl-7B Claude 4.5 Sonnet Doubao-1.5-Thinking OpenAI CUA

Builds on: OSWorld WebJudge AgentNet GUI-Reflection ReAct

Closest papers by meaning

• OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models (CVPR 2026 · similarity 0.87)
• GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions (NeurIPS 2026 · similarity 0.87)
• D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies (AAAI 2026 · similarity 0.86)
• DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents (NeurIPS 2026 · similarity 0.86)
• OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.85)
• UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents (NeurIPS 2025 · similarity 0.85)