Research map / Multi-agent systems and games / LLM agents

PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PAGE Bench (4,906 GeoGebra geometry construction problems with GUI trajectories) and PAGER, a dependency-structured planner plus pixel-level executor trained with SFT and precision-aligned RL on Qwen3-VL-8B for point-precise geometric GUI drawing.

Problem

Existing GUI agents succeed under region-tolerant interaction, but precise geometric construction on a continuous canvas requires point-level accuracy, and small coordinate errors cascade through dependent geometric objects.

Approach

Factorize geometric drawing into a dependency-aware sub-task planner and a pixel-grounded GUI executor, train with SFT on verified GeoGebra trajectories, then apply RL with rewards for action-type correctness, parameter accuracy, and rendered geometric validity, with an SFT KL anchor.

Key results

• PAGER achieves Overall 29.52 and Task Success 23.78 on PAGE Bench, versus 24.36 Overall for the best general baseline Gemini-3.1-Pro (Task Success 5.82).
• Best GUI-specialized agent reaches only 16.18 Step Success, while PAGER reaches 62.20.
• Removing the parameter-accuracy reward lowers Overall from 20.47 (SFT) to 20.07 and removing the action-type reward yields 24.52 Overall, versus 29.52 for the full model.

Surprising findings

• Closed-source VLMs such as Claude-Sonnet-4.6 reach about 95.85 action accuracy yet only 1.11 task success, indicating operation semantics are not the bottleneck.
• Without the parameter-accuracy reward, RL gains little beyond SFT and Overall slightly drops (20.47 to 20.07).

Limitations

• Scope limited to GeoGebra-style planar construction; other precision-sensitive interfaces need new action grammars and validity rules.
• Dataset is built with LLM-based authoring, screening, and filtering, so it relies on model-generated annotations.

Experimental setup

Datasets: PAGE Bench public K-12 multimodal geometry resources

Benchmarks: PAGE Bench

Metrics: action accuracy parameter accuracy step success task success middle process score final result score overall score human correlation

Baselines: Qwen3-VL-8B DeepSeek-VL2 GLM-4.5V InternVL2.5-8B KimiVL-A3B MiniCPM-V-2.6 LLaVA-NeXT-8B Claude-Sonnet-4.6 GPT-5.4 Qwen3.6-Plus Gemini-3.1-Pro UI-TARS

Builds on: Qwen3-VL GRPO

Closest papers by meaning

• HATS: Hardness-Aware Trajectory Synthesis for GUI Agents (CVPR 2026 · similarity 0.89)
• GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.88)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.88)
• GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents (NeurIPS 2025 · similarity 0.87)
• Learning GUI Grounding with Spatial Reasoning from Visual Feedback (ICML 2026 · similarity 0.87)
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.87)