Research map / Large language models / Reasoning and chain of thought

UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction as Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Instruction-as-Reasoning, an SFT+GRPO framework that treats multi-perspective instructions (appearance, function, location, intent) as reasoning pathways for GUI grounding, yielding UI-Ins-7B/32B with SOTA results on five grounding benchmarks.

Problem

GUI grounding maps natural-language instructions to UI elements, but existing datasets have flawed instructions (23.3% in a manual sample) and models are trained on a single narrow instruction style, limiting grounding accuracy and RL exploration.

Approach

Treat instructions as dynamic reasoning pathways: SFT teaches the model to generate a rewritten instruction from a chosen perspective before predicting a coordinate, then GRPO with a point-in-box reward lets it select or compose perspectives, which also stabilizes SFT+RL against policy collapse.

Key results

• UI-Ins-32B scores 87.3% on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2.
• UI-Ins-7B reaches 74.1% success on AndroidWorld with a GPT-5 planner, above Gemini 2.5 Computer Use (69.7%) and UI-TARS-2 (73.3%).
• Ablation: SFT+RL with instruction-as-reasoning gives 52.2% on ScreenSpot-Pro for Qwen2.5-VL-7B versus 47.5% for RL without it; free-form reasoning in RL lowers UI-TARS-1.5-7B from 50.1 to 46.9 on ScreenSpot-Pro.

Surprising findings

• Free-form reasoning during RL degrades or does not improve grounding accuracy, while instruction-as-reasoning improves it.
• Standard SFT followed by RL degrades performance (e.g., JEDI-7B drops from 39.5 to 34.5 on ScreenSpot-Pro), while instruction-as-reasoning SFT before RL avoids this collapse.

Limitations

• Instruction-quality analysis relies on a manual inspection of 1,909 samples and GPT-4.1 verification, and the pipeline still leaves an estimated error rate below 8%.
• Error analysis shows failures from missing domain knowledge, weak layout understanding, and visual ambiguity with distractor icons.
• Evaluation of the agent relies on a GPT-5 planner, so the online result depends on a closed-source model.

Experimental setup

Datasets: OS-Atlas OmniAct Android Control AMEX AgentNet Widget Captioning

Benchmarks: MMBench-GUI L2 UI-I2E-Bench ScreenSpot-Pro ScreenSpot-V2 Showdown AndroidWorld

Metrics: point-in-box accuracy task success rate

Baselines: GTA1-7B GUI-Actor-7B SE-GUI-7B GUI-G2-7B InfiGUI-G1-7B UI-TARS-1.5-7B Qwen2.5-VL-7B Qwen2.5-VL-32B GTA1-32B JEDI-7B

Builds on: GRPO Qwen2.5-VL OmniParser V2 Phi-Ground

Closest papers by meaning

• GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.92)
• Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.92)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis (NeurIPS 2025 · similarity 0.91)
• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Test-Time Reinforcement Learning for GUI Grounding via Region Consistency (AAAI 2026 · similarity 0.90)
• Learning GUI Grounding with Spatial Reasoning from Visual Feedback (ICML 2026 · similarity 0.89)