Research map / Multi-agent systems and games / LLM agents
GTA1: GUI Test-time Scaling Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GTA1, a two-stage GUI agent pairing test-time scaling of planner action proposals (judge selects among K samples) with an RL-trained (GRPO, click reward, no thinking) coordinate grounding model, reaching SOTA on ScreenSpot-Pro/V2/OSWorld-G and OSWorld agent tasks.
Problem
GUI agents face planning ambiguity over many valid action sequences (with no lookahead, irreversible actions) and imprecise grounding of actions in high-resolution, complex interfaces.
Approach
Sample K candidate action proposals per step from the planner, have a multimodal LLM judge pick the best, and ground clicks with an RL model trained with a binary click reward (point inside the annotated box) without chain-of-thought, after IoU-based data cleaning with OmniParser.
Key results
- GTA1-7B reaches 50.1% on ScreenSpot-Pro, above UGround-72B's 34.5%; GTA1-32B reaches 95.2% on ScreenSpot-V2 and 72.2% on OSWorld-G.
- GTA1-7B with o3 planner achieves 45.2% on OSWorld (100 steps), above CUA o3's 42.9% at 200 steps; GTA1-32B w/ GPT-5 reaches 63.4% on OSWorld-Verified.
- Click-only reward outperforms adding IoU or format (thinking) rewards on ScreenSpot-Pro (50.1 vs 42.2-46.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add IoU reward and format (thinking) reward alongside click reward (ablation) | GTA1 grounding model · ScreenSpot-Pro | accuracy: 44.5 vs 50.1 click-only | Hurt |
| add format (thinking) reward to click reward (ablation) | GTA1 grounding model · OSWorld-G | accuracy: 67.0 vs 67.7 click-only | Hurt |
| add IoU reward to click reward (ablation) | GTA1 grounding model · ScreenSpot-V2 | accuracy: 89.2 vs 92.4 click-only | Hurt |
| add format (thinking) reward to click reward (ablation) | GTA1 grounding model · ScreenSpot-V2 | accuracy: 93.2 vs 92.4 click-only | Mixed |
| test-time scaling for planning: sample K candidates per step and select with judge model (o3, K=8) (ablation) | GTA1-7B agent with o3 planner · OSWorld (50 steps) | success rate: 43.4 vs 41.3 at K=1 | Helped |
| test-time scaling (K=32 candidates, 15 steps) (ablation) | UI-TARS-1.5-7B · OSWorld | success rate: 15-step K=32 outperforms 100-step K=1 baseline | Helped |
| use click-only reward (no thinking, no IoU) for grounding RL (ablation) | GTA1-7B grounding · ScreenSpot-Pro | accuracy: 50.1 vs 44.5 with all rewards | Helped |
| data cleaning by discarding samples whose annotated box has IoU < 0.3 with OmniParser boxes | GTA1 grounding training data · ScreenSpot-Pro | accuracy | No clear effect |
Surprising findings
- Explicit chain-of-thought reasoning does not help GUI grounding in static benchmarks and can hurt accuracy when used with RL.
- A two-stage agent (planner plus grounding model) can outperform end-to-end native GUI agents on OSWorld with fewer steps.
Limitations
- Struggles in certain scenarios such as selecting custom foregrounds and backgrounds in image editing (GMIP).
- Relies on large proprietary planners (o3, GPT-5) and judges for best agent results.
- Some baseline numbers evaluated by the authors may change with improved evaluation scripts.
Experimental setup
Datasets: Aria-UI-Web OmniACT UI Vision Widget Caption OS-Atlas-Desktop AndroidControl ScreenSpot-Pro ScreenSpot-V2 OSWorld-G OSWorld WindowsAgentArena
Benchmarks: ScreenSpot-Pro ScreenSpot-V2 OSWorld-G OSWorld OSWorld-Verified WindowsAgentArena
Metrics: grounding accuracy task success rate
Baselines: UI-TARS-1.5-7B UI-TARS-72B Jedi-7B SE-GUI-7B GUI-G1-3B Qwen2.5-VL-72B OpenCUA-32B Seed-1.5-VL CUA o3 Agent S2.5
Builds on: GRPO DeepSeek-R1-Zero OmniParser UI-TARS-1.5 OpenCUA
Closest papers by meaning
- SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.91)
- Test-Time Reinforcement Learning for GUI Grounding via Region Consistency (AAAI 2026 · similarity 0.91)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.89)
- Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.89)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.