Research map / Large language models / Agents, code and math
Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a pre-operative critic (GUI-Critic-R1, 7B Qwen2.5-VL) trained with RFT cold-start and suggestion-aware GRPO on newly built GUI-Critic-Train/Test data to diagnose GUI actions before execution.
Problem
MLLM-based GUI agents make step-level errors that cumulate or are irreversible (e.g., deletions, payments), and existing MLLMs struggle to critique actions before execution; no training or test data existed for GUI pre-critics.
Approach
A pre-operative critic that judges an intended action before execution and gives a corrective suggestion, trained via RFT cold-start followed by GRPO with an added suggestion reward (LLM-judged similarity of suggestion to annotation), plus a reasoning-bootstrapping pipeline for CoT critique data.
Key results
- On GUI-I, GUI-Critic-R1 reaches 69.20% critic accuracy and 52.43% suggestion accuracy, vs 54.88% and 43.14% for the Qwen2.5-VL-7B base.
- On AndroidWorld with the M3A agent, pre-critic raises success rate from a 22.4% baseline to 27.6%, with EAR of 31.8%, above GPT-4o pre-critic (22.4 SR, 26.1 EAR).
- Ablation: removing the suggestion reward drops GUI-I suggestion accuracy from 52.43% to 49.24% (with RFT and other rewards kept) and GRPO alone without RFT cold-start yields only 54.38-67.98% critic accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GUI-Critic-R1 pre-critic | Qwen2.5-VL-7B GUI agent · AndroidWorld | success rate (SR): 22.4 -> 27.6 | Helped |
| add GUI-Critic-R1 pre-critic | Qwen2.5-VL-7B GUI agent · AndroidWorld | Efficiency Advantage Rate (EAR) | Helped |
| train GUI-Critic-R1 with the proposed approach | Qwen2.5-VL-7B · GUI-I | critic accuracy: 54.88 -> 69.20 | Helped |
| train GUI-Critic-R1 with the proposed approach | Qwen2.5-VL-7B · GUI-S | critic accuracy: 57.02 -> 58.77 | No clear effect |
| replace random negative decisions with Negative Operations Sampling (ablation) | GUI-Critic-R1 training data · GUI-I | critic accuracy: 50.46 -> 69.20 | Helped |
| replace random negative decisions with Negative Operations Sampling (ablation) | GUI-Critic-R1 training data · GUI-I | suggestion accuracy: 1.22 -> 52.43 | Helped |
| add data filtering instead of random sample selection (ablation) | GUI-Critic-R1 training data · GUI-I | critic accuracy: 67.23 -> 69.20 | Helped |
| add GUI Critique Generation to the RFT cold-start data (ablation) | GUI-Critic-R1 · GUI-I | critic accuracy: 67.84 -> 69.20 | Helped |
| add RFT cold-start before GRPO (ablation) | GUI-Critic-R1 · GUI-I | suggestion accuracy: 49.24 -> 52.43 | Helped |
| add suggestion reward to S-GRPO (ablation) | GUI-Critic-R1 · GUI-I | suggestion accuracy: 47.71 -> 52.43 | Helped |
| add suggestion reward to S-GRPO (ablation) | GUI-Critic-R1 · GUI-S | suggestion accuracy: 40.35 -> 47.37 | Helped |
| increase suggestion reward weight excessively (ablation) | GUI-Critic-R1 S-GRPO · GUI-I | critic accuracy | Hurt |
Surprising findings
- GRPO alone without an RFT cold-start was insufficient to activate critic abilities, since the base model cannot produce good samples randomly.
- GPT-4o as a pre-critic sometimes produced misleading critiques, lowering agent performance on AndroidWorld due to weak GUI common-sense.
Limitations
- Single-step critic using screenshot and semantic history only; no trajectory-level critic.
- Evaluated only with a 7B model; lighter or other scales not explored.
- Suggestion accuracy depends on an LLM judge (Qwen2.5-VL-72B) and binary similarity.
Experimental setup
Datasets: AITZ AMEX Odyssey AITW GUICourse
Benchmarks: GUI-Critic-Test (GUI-I, GUI-S, GUI-W) AndroidWorld
Metrics: critic accuracy suggestion accuracy success rate (SR) efficiency advantage rate (EAR)
Baselines: Claude-3.5 GPT-4o Gemini-2.0-Flash Deepseek-VL2-7B InternVL2.5-8B InternVL2.5-8B-MPO Qwen2.5-VL-7B Qwen2.5-VL-72B
Builds on: GRPO DeepSeek-R1 Qwen2.5-VL M3A (AndroidWorld) Critic-V
Closest papers by meaning
- OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models (CVPR 2026 · similarity 0.89)
- GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior (NeurIPS 2025 · similarity 0.88)
- ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.86)
- VideoGUI: A Benchmark for GUI Automation from Instructional Videos (NeurIPS 2024 · similarity 0.86)
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.85)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.