Research map / Multi-agent systems and games / LLM agents

Grounding Computer Use Agents on Human Demonstrations

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GroundCUA, a human-annotated desktop grounding dataset (87 apps, 56K screenshots, 3.56M elements), and GroundNext 3B/7B grounding models trained with SFT on 700K instructions followed by RL on 10K samples.

Problem

GUI grounding for computer-use agents is hard on desktop software, where high-resolution dense screens and small icons are underrepresented in existing automatically collected datasets.

Approach

Build a dense, expert human-demonstration-based desktop grounding dataset with human-verified bounding boxes and labels, generate diverse direct/functional/spatial instructions from it, and train Qwen2.5-VL models with SFT then a simple discrete distance-based RL reward (RLOO).

Key results

• GroundNext-7B (SFT) reaches 69.2 average across five grounding benchmarks, ahead of JEDI-7B (56.1) on SFT-only comparison, using 700K instructions.
• GroundNext-7B (RL) reaches 70.5 average and GroundNext-3B (RL) 68.4, with RL giving modest gains over SFT.
• GroundNext-3B (RL) with o3 planner scores 50.6 overall on OSWorld-Verified, comparable to JEDI-7B w/ o3 (51.0) and above OpenCUA-72B (46.1).

Surprising findings

• RL gains are smallest for the model SFT-trained on GroundCUA, suggesting stronger SFT leaves fewer errors for RL to correct.
• Functional instructions give the best average among instruction types in ablation, while spatial instructions perform worst (47.5 avg).

Limitations

• Limited scale and compute for training experiments.
• Simple discrete reward; more sophisticated reward designs not explored.
• Weaker cross-domain generalization on web splits of SSv2 than mobile.

Experimental setup

Datasets: GroundCUA UI-Vision OS-Atlas desktop Aguvis UGround JEDI

Benchmarks: ScreenSpot-Pro OSWorld-G MMBench-GUI (L2) ScreenSpot-v2 UI-Vision OSWorld-Verified

Metrics: accuracy success rate

Baselines: Qwen2.5-VL-3B Qwen2.5-VL-7B PhiGround-4B-7C JEDI-3B JEDI-7B GUI-Actor-3B GUI-Actor-7B OS-Atlas-7B UGround-V1-7B Aguvis-7B InfiGUI-G1-3B GTA1-7B

Builds on: Qwen2.5-VL RLOO UI-Vision LlamaFactory DeepSeek-R1

Closest papers by meaning

• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis (NeurIPS 2025 · similarity 0.90)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.89)
• GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.89)
• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction as Reasoning (ICLR 2026 · similarity 0.88)
• Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.88)