Research map / Reinforcement learning / RL from feedback
SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SE-GUI, an RL-based GUI grounding method combining a filtered 3k seed dataset, a dense point reward within GRPO, and attention-guided self-evolutionary reinforcement fine-tuning, reaching 47.3% on ScreenSpot-Pro with a 7B model.
Problem
GUI grounding (mapping natural language instructions to precise interface elements) is hard in high-resolution professional settings, and SFT needs large diverse data and generalizes poorly; binary rewards in RL are sparse and noisy grounding data hurts training.
Approach
Combines three elements: a three-stage data filter (instruction quality, bounding box accuracy, task difficulty) yielding 3k samples; a continuous point reward that gives partial credit by distance to the ground-truth box center within GRPO; and an iterative self-evolution loop where attention maps from the previous-stage model zero out the loss for samples whose attention does not peak in or favo
Key results
- SE-GUI-7B reaches 47.3% average on ScreenSpot-Pro, outperforming UI-TARS-72B (38.1%) at 7B scale with 3k training samples.
- SE-GUI-7B achieves 88.2% on ScreenSpot and 90.3% on ScreenSpot-v2 (Avg.).
- Self-evolution stages improve ScreenSpot-Pro from 39.97% (stage 1) to 46.55% (stage 3), with stage 4 converging.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dense point rewards instead of sparse rewards with filtered data (ablation) | SE-GUI-3B · ScreenSpot-Pro | average accuracy: 35.8 -> 40.0 | Helped |
| Filter the training data and use dense rewards instead of original data and sparse rewards (ablation) | SE-GUI-3B · ScreenSpot-Pro | average accuracy: 27.1 -> 40.0 | Helped |
| Use filtered data instead of original unfiltered data with sparse rewards (ablation) | SE-GUI-3B · ScreenSpot-Pro | average accuracy: 27.1 -> 35.8 | Helped |
| Set reward ratio to 1:2 and KL coefficient to 0.004 instead of 1:1 and 0.04 (ablation) | SE-GUI-3B · ScreenSpot-Pro | average accuracy: 24.85 -> 34.28 | Helped |
| Set attention filtering threshold to 0.2 instead of 0.4 (ablation) | SE-GUI-7B · ScreenSpot-Pro | average accuracy: 42.95 -> 44.78 | Helped |
| Set attention filtering threshold to 0.2 instead of 0.1 (ablation) | SE-GUI-7B · ScreenSpot-Pro | average accuracy: 43.51 -> 44.78 | Helped |
| Continue self-evolution from stage 1 through stage 3 using previous-stage attention maps (ablation) | SE-GUI-7B · ScreenSpot-Pro | accuracy: 39.97 -> 46.55 | Helped |
| Continue self-evolution to stage 4 rather than stop at stage 3 (ablation) | SE-GUI-7B · ScreenSpot-Pro | accuracy | No clear effect |
| Use SE-GUI-7B instead of UI-TARS-72B | GUI grounding models · ScreenSpot-Pro | average accuracy: 38.1 -> 47.3 | Helped |
| Use SE-GUI-7B instead of Aguvis-7B | GUI grounding models · ScreenSpot | accuracy (%): 84.4 -> 88.2 | Helped |
| Use SE-GUI-7B instead of Aguvis-7B | GUI grounding models · ScreenSpot-v2 | accuracy (%): 87.3 -> 90.3 | Helped |
| Use SE-GUI-7B grounding instead of UGroundV1-7B grounding | GPT-4o planner · AndroidControl-High | SR: 48.4 -> 52.8 | Helped |
Surprising findings
- Filtering the dataset and using dense rewards each help, but on the unfiltered data the dense reward alone gave a larger gain than on filtered data in one ablation table.
- The 3B variant with dense reward reaches 35.9% on ScreenSpot-Pro, close to UI-TARS-7B's 35.7%.
Limitations
- Only 3B and 7B scales explored due to hardware limits.
- 7B model input resolution capped at 2M pixels, possibly losing fine detail.
- Self-evolution gains saturate after about three stages.
Experimental setup
Datasets: ShowUI UGround AriaUI SE-GUI-3k
Benchmarks: ScreenSpot ScreenSpot-v2 ScreenSpot-Pro AndroidControl-High AndroidControl-Low OmniACT
Metrics: grounding accuracy step success rate (SR) action score (AS)
Baselines: UI-TARS-72B UI-TARS-7B UGround-V1-7B Qwen2.5-VL-7B Qwen2.5-VL-3B OS-Atlas-7B GUI-R1-7B UI-R1-3B GPT-4o Claude Computer Use
Builds on: GRPO DeepSeek-R1 Qwen2.5-VL UI-R1 GUI-R1
Closest papers by meaning
- GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.93)
- GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning (ICLR 2026 · similarity 0.93)
- Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.92)
- GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- GTA1: GUI Test-time Scaling Agent (ICLR 2026 · similarity 0.92)
- GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.