Research map / Multi-agent systems and games / LLM agents
BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BTL (Blink-Think-Link), a structured output paradigm with ROI "blink" annotations and a rule-based process-plus-outcome reward, trained via GRPO to build the BTL-UI GUI agent on Qwen2.5-VL-3B/7B.
Problem
Existing rule-based RFT GUI agents deviate from human-GUI interaction patterns and reward only final outcomes, lacking process-level supervision of attention and intermediate steps.
Approach
Decomposes each GUI step into blink (ROI bounding-box output), think (reasoning), and link (action) phases, with an automated ROI annotation pipeline and a rule-based BTL reward combining dual format reward, Hungarian-matched blink reward, and all-or-nothing link reward, optimized with GRPO.
Key results
- BTL-UI-7B reaches 87.2% average on ScreenSpot and 89.1% on ScreenSpot-V2, above Qwen2.5-VL-7B (84.8%) on ScreenSpot.
- BTL-UI-7B achieves 88.0% SR on AndroidControl-Low and 69.2% SR on AndroidControl-High, above GUI-R1-7B (66.5% and 51.7%).
- BTL-UI-7B scores 33.7% average on ScreenSpot-Pro, versus 29.0% for Qwen2.5-VL-7B; BTL-UI-3B scores 27.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train BTL-UI-7B with Blink-Think-Link RFT instead of zero-shot Qwen2.5-VL-7B | Qwen2.5-VL-7B · AndroidControl-High | SR: 47.1 -> 69.2 | Helped |
| train BTL-UI-7B with Blink-Think-Link RFT instead of zero-shot Qwen2.5-VL-7B | Qwen2.5-VL-7B · AndroidControl-Low | SR: 62.5 -> 88.0 | Helped |
| train BTL-UI-7B with Blink-Think-Link RFT instead of zero-shot Qwen2.5-VL-7B | Qwen2.5-VL-7B · GUI-Odyssey | SR: 34.4 -> 65.4 | Helped |
| train BTL-UI-7B with Blink-Think-Link RFT instead of zero-shot Qwen2.5-VL-7B | Qwen2.5-VL-7B · ScreenSpot | Avg. accuracy: 84.8 -> 87.2 | Helped |
| train BTL-UI-7B with Blink-Think-Link RFT instead of zero-shot Qwen2.5-VL-7B | Qwen2.5-VL-7B · ScreenSpot-V2 | Avg. accuracy | Helped |
| train BTL-UI-3B with Blink-Think-Link RFT instead of UI-R1-3B | UI-R1-3B · ScreenSpot-Pro | Avg. accuracy: 17.8 -> 27.1 | Helped |
| add Blink Data to SFT training (ablation) | BTL-UI-7B (SFT) · AndroidControl-High | SR: 60.6 -> 65.6 | Helped |
| add Blink Data and BTL Reward to RFT (ablation) | BTL-UI-7B (RFT only) · AndroidControl-High | SR: 65.4 -> 69.2 | Helped |
| add Blink Data to SFT training (ablation) | BTL-UI-7B (SFT) · AndroidControl-High | Type: 79.4 -> 86.4 | Helped |
| add Blink Data to SFT training (ablation) | BTL-UI-7B (SFT) · AndroidControl-High | GR: 63.9 -> 69.9 | Helped |
| increase Blink ROI count from λ=1 to λ=5 (ablation) | BTL-UI-7B · AndroidControl-High | SR: 66.6 -> 69.2 | Helped |
| increase Blink ROI count from λ=5 to λ=6 (ablation) | BTL-UI-7B · AndroidControl-High | SR: 69.2 -> 69.2 | No clear effect |
Surprising findings
- Blink data also helps SFT: SFT with blink data raises SR on AndroidControl-High from 60.6% to 65.6%.
- UI-TARS-7B remains stronger than BTL-UI on several grounding and planning benchmarks, which the authors attribute to large-scale pretraining (50B tokens).
Limitations
- The blink output lengthens sequences, adding computational overhead.
- AndroidControl is offline and contains labeling noise that can mislead evaluation of reasoning.
- Overall grounding remains slightly below UI-TARS-7B, which uses large-scale pretraining.
Experimental setup
Datasets: ShowUI-Web ShowUI-Desktop AndroidControl GUI-Odyssey
Benchmarks: ScreenSpot ScreenSpot-V2 ScreenSpot-Pro AndroidControl GUI-Odyssey
Metrics: click accuracy action type accuracy grounding accuracy (GR) step success rate (SR)
Baselines: UI-R1 GUI-R1 UI-TARS OS-Atlas-Pro Aguvis Aria-UI SeeClick GPT-4o Qwen2.5-VL
Builds on: GRPO DeepSeek-R1 Qwen2.5-VL OS-Atlas ms-swift
Closest papers by meaning
- GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.87)
- AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · similarity 0.87)
- History-Aware Reasoning for GUI Agents (AAAI 2026 · similarity 0.86)
- See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles (CVPR 2026 · similarity 0.86)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.86)
- SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.