Research map / Multi-agent systems and games / LLM agents

Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agents

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Co-EPG, a self-iterative framework where a planning model and a grounding model co-evolve via GRPO with grounding-based rewards (C-DREM ensemble) and self-generated data, reaching SOTA on Multimodal-Mind2Web and AndroidControl.

Problem

GUI agents combine planning and grounding, but existing methods optimize these components independently and rely heavily on large synthetic datasets.

Approach

A positive feedback loop where the planning model is trained with GRPO using rewards from grounding-model accuracy (ensembled via confidence- and prior-weighted C-DREM), while the improved grounding model is re-trained on high-quality data distilled from the planner's verified plans.

Key results

• Co-EPG-Web-7B reaches 58.4% average Step SR on Multimodal-Mind2Web, above AGUVIS-7B (57.2%) and Explorer-7B (54.3%).
• Co-EPG-Mob-7B reaches 83.1% average Step Acc on AndroidControl, 1.4 points above UI-TARS-7B (81.7%).
• Co-EPG-Web-7B uses only 6,862 labeled steps versus 283,500 for AGUVIS-7B (2.42%).

Surprising findings

• Co-EPG-Web-7B surpasses AGUVIS-7B using only 2.42% of its labeled step data.
• Removing GRPO and relying only on SFT across iterations still improves performance but significantly underperforms the full framework.

Limitations

• Relies on large external VLMs (Qwen2.5-VL-32B/72B) as reward sources, adding inference cost.
• Evaluated mainly on benchmark data; the authors note that combining with other data synthesis techniques is left for future work.
• Reported gains over the strongest baselines are small (e.g., 1.4 points on AndroidControl).

Experimental setup

Datasets: Multimodal-Mind2Web AndroidControl OmniACT

Benchmarks: Multimodal-Mind2Web AndroidControl OmniACT

Metrics: Ele.Acc Op.F1 Step SR Step Acc Action Score

Baselines: Explorer AGUVIS AgentTrek SeeClick OmniParser UI-TARS InfiGUI-R1 OS-Atlas GUI-R1 UI-R1 UGround

Builds on: GRPO Qwen2.5-VL MS-SWIFT AGUVIS

Closest papers by meaning

• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.88)
• Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.88)
• GTA1: GUI Test-time Scaling Agent (ICLR 2026 · similarity 0.88)
• SE-GA: Memory-Augmented Self-Evolution for GUI Agents (ICML 2026 · similarity 0.87)
• GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.87)