Research map / Multi-agent systems and games / LLM agents
SpiritSight Agent: Advanced GUI Agent with One Look
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SpiritSight, a vision-only end-to-end GUI agent on InternVL2 (2B/8B/26B) pre-trained on a 5.73M-sample multi-level GUI dataset (GUI-Lasagne) and uses Universal Block Parsing to fix coordinate ambiguity under dynamic high-resolution inputs.
Problem
Vision-only GUI agents have low element-grounding accuracy, and dynamic high-resolution tiling in VLMs creates locational ambiguity when block-flattened coordinates are predicted as global coordinates.
Approach
Pre-train on a large hierarchical GUI dataset (text/icon grounding, function grounding, CoT navigation) and replace global coordinates with block-specific coordinates [block index, x', y'] plus 2D block-wise position embeddings, so the mapping from input to output becomes injective; global coordinates are recovered in post-processing.
Key results
- SpiritSight-26B reaches 54.7% cross-task Step SR and 49.2% cross-domain Step SR on Multimodal-Mind2Web, outperforming all non-candidate-based methods listed.
- SpiritSight-8B scores 68.3% on ScreenSpot web, 68.4% mobile, and 62.9% desktop, beating SeeClick (44.1/65.0/51.1).
- SpiritSight-2B achieves 96.1% hit rate on a custom 3,700-pair text2bbox test set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Universal Block Parsing (UBP) instead of baseline block parsing (ablation) | InternVL GUI agent · Multimodal-Mind2Web | Ele.Acc | Helped |
| use UBP instead of baseline block parsing (ablation) | InternVL GUI agent · Multimodal-Mind2Web | Op.F1 | No clear effect |
| add 2D-BPE to UBP (ablation) | InternVL GUI agent · Multimodal-Mind2Web | performance | Helped |
| remove GUI-Lasagne level-1 data (ablation) | SpiritSight-8B · Multimodal-Mind2Web | Step SR | Hurt |
| remove GUI-Lasagne level-2 data (ablation) | SpiritSight-8B · Multimodal-Mind2Web | Step SR | Hurt |
| remove GUI-Lasagne level-3 data (ablation) | SpiritSight-8B · Multimodal-Mind2Web | Step SR | Hurt |
| add CoT construction to level-3 training data (ablation) | SpiritSight-8B · Multimodal-Mind2Web | Step SR | Helped |
| clean level-3 training data (ablation) | SpiritSight-8B · Multimodal-Mind2Web | Step SR | No clear effect |
| increase GUI-Lasagne pre-training dataset size (ablation) | SpiritSight · Multimodal-Mind2Web | performance | Helped |
| increase model size (ablation) | SpiritSight · Multimodal-Mind2Web | performance | Helped |
| fine-tune on English-only rather than English and Chinese data (ablation) | SpiritSight-Base-8B · GUIAct web-multi Chinese test | Step SR: 49.3% -> 24.5% | Hurt |
Surprising findings
- Training on only GUI-Lasagne level-1 data already outperforms three models trained on SeeClick data, and SeeClick retrained on InternVL with full-parameter training performs worst.
- Data cleaning of AitW with GPT-4o had a relatively small effect on navigation performance during pre-training.
Limitations
- Vision-only agents require access to screenshots that may contain sensitive personal information.
- GPT-4o cleaning of discarded steps has only 76.3% true negative rate on human check.
Experimental setup
Datasets: GUI-Lasagne CommonCrawl AitW AITZ SeeClick training data Multimodal-Mind2Web GUIAct Ureader-Instruction GUIChat WebSRC ScreenQA RICO
Benchmarks: Multimodal-Mind2Web ScreenSpot AMEX GUIAct AndroidControl GUI-Odyssey
Metrics: Ele.Acc Op.F1 Step SR click accuracy action matching score step accuracy
Baselines: SeeClick OmniParser CogAgent SeeAct MiniCPM-GUI OdysseyAgent GPT-4o
Builds on: InternVL2 SeeClick 2D-BPE
Closest papers by meaning
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.89)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.89)
- Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction (ICML 2025 · similarity 0.88)
- GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.87)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.86)
- Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.