Research map / Multi-agent systems and games / LLM agents

Continual GUI Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Continual GUI Agents, a domain- and resolution-shift continual learning setting, and GUI-AiF, an RFT (GRPO-based) method adding point-diversity and region-separation rewards (APR-iF, ARR-iF) to mitigate over-adaptation of GUI grounding.

Problem

GUI grounding agents trained on static datasets degrade when new GUI data arrives with shifted domains (mobile, desktop, web) or resolutions (1080p to 4K), since they over-adapt to fixed coordinates and element scales.

Approach

Adds two auxiliary reward terms to GRPO-style reinforcement fine-tuning: APR-iF rewards spatial variance of predicted center points, and ARR-iF rewards Bhattacharyya-distance separation among Gaussian-modeled predicted boxes, encouraging diverse interaction points and region scales instead of over-fitting the current task.

Key results

• GUI-AiF reaches 81.7% average on ScreenSpot-V1 and 83.5% on ScreenSpot-V2 after the Mobile→Desktop→Web sequence with Qwen2.5-VL-3B, versus 78.1%/81.2% for the GUI-G2 base reported in the paper's table (GUI-G2 row: 77.1/80.3).
• On ScreenSpot-Pro normal→high resolution, GUI-AiF averages 19.0% versus 16.7% for GUI-G2 under the same sequence.
• Removing the KL term degrades continual performance, with the no-KL variant reaching 77.1% on SSv1 and 79.6% on SSv2 after the full domain sequence.

Surprising findings

• The authors report an inverse relationship between the new diversity rewards and the standard RFT point/coverage rewards, suggesting conflicting objectives.
• Performance on text-based targets is consistently higher than on icon targets across domains and resolutions, which the authors attribute to a bias toward OCR capability.

Limitations

• Limited to a 3B model, three training datasets, and three benchmarks due to compute constraints.
• Only domain and resolution shifts are studied; layout, appearance, and language shifts are left to future work.
• Hyperparameter alpha/gamma sensitivity is reported, and the paper notes the upper bounds come from baselines trained on larger data.

Experimental setup

Datasets: Widget Captioning ShowUI-web OmniACT

Benchmarks: ScreenSpot-V1 ScreenSpot-V2 ScreenSpot-Pro

Metrics: accuracy

Baselines: SeeClick GUI-Actor SE-GUI† GUI-G2 InfiGUI-R1 Qwen2.5VL-3B GPT-4o Claude Computer Use

Builds on: GRPO GUI-G2 SE-GUI InfiGUI-R1

Closest papers by meaning

• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.89)
• Test-Time Reinforcement Learning for GUI Grounding via Region Consistency (AAAI 2026 · similarity 0.88)
• GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning (ICLR 2026 · similarity 0.88)
• GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.87)