Research map / Multi-agent systems and games / LLM agents

Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI Automation

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CES, a Coordinator-Executor-State Tracker multi-agent framework where the Coordinator and State Tracker are trained with staged execution-feedback GRPO while a frozen executor provides rewards, improving long-horizon GUI automation.

Problem

GUI agents struggle on long-horizon tasks due to responsibility coupling between high-level planning and low-level execution in one policy, and lack of task-state awareness that leads to progress loss.

Approach

Decouple a GUI agent into a trained high-level Coordinator (planning, atomic instruction generation) and a trained State Tracker (natural-language state compression), optimized with GRPO using rewards from a frozen Executor's actions, in two stages; the trained modules plug into any Executor.

Key results

• On GUI-Odyssey, CES with GUI-R1-7B executor reaches 79.24 Type, 63.82 GR, 53.69 SR, versus 65.49/43.64/38.79 for GUI-R1-7B alone.
• Applied to GUI-Owl-32B on GUI-Odyssey, CES raises SR from 39.60 to 56.75.
• Ablation: removing the Coordinator drops GUI-Odyssey SR from 53.69 to 39.15; removing the State Tracker drops it to 42.52.

Surprising findings

• Prompting GPT-5 as Coordinator and State Tracker yields only minor and sometimes degraded gains, while the trained modules help more.
• CES-P (prompting the same model in all roles) degraded UI-R1-3B's SR on GUI-Odyssey by 18.05 points.

Limitations

• Executor is frozen and its perception errors remain the main bottleneck after CES.
• RL training data and evaluation both drawn from GUI-Odyssey-family benchmarks, limiting generality evidence.
• Relies on offline ground-truth actions for rewards rather than online environment success.

Experimental setup

Datasets: GUI-Odyssey AITZ AMEX

Benchmarks: AITZ AMEX GUI-Odyssey

Metrics: Type accuracy GR (grounding) accuracy SR (success rate)

Baselines: GPT-4o Qwen2.5-VL-7B OS-Atlas-7B UI-R1-3B GUI-Owl-7B SWIRL GUI-R1-7B GPT-5 multi-agent GUI-Owl-32B

Builds on: GRPO DeepSeek-R1 GUI-R1

Closest papers by meaning

• GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.89)
• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• History-Aware Reasoning for GUI Agents (AAAI 2026 · similarity 0.87)
• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.87)
• GTA1: GUI Test-time Scaling Agent (ICLR 2026 · similarity 0.87)