Research map / Vision-language and multimodal / Multimodal LLMs
Attention-Driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models Without Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAG, a tuning-free GUI grounding method that aggregates selected self-attention and cross-attention maps from MiniCPM-Llama3-V 2.5 to localize text and icons without fine-tuning.
Problem
GUI grounding (locating text or icon elements from a screenshot and text query) typically requires fine-tuning MLLMs on specialized GUI data; pretrained MLLMs localize poorly when asked to output boxes directly.
Approach
Prompt the MLLM to generate a description of the target element, select top-K attention heads per text token by their total attention to visual query tokens, propagate attention through the cross-attention compressor to image patches, and threshold the aggregated map to get the predicted location.
Key results
- On ScreenSpot, TAG on MiniCPM-Llama3-V 2.5 reaches 54.8% average accuracy without fine-tuning, versus 36.0% for MiniCPMV2.5 direct prediction and 53.4% for SeeClick.
- On the OCG dataset across 10 aspect ratios, TAG averages 84.5% versus 48.1% for MiniCPMV2.5 and 60.2% for SeeClick.
- On Mind2Web element accuracy, TAG averages 24.3%, matching SeeClick fine-tuned on Mind2Web (24.3%) and exceeding MiniCPMV2.5 (15.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TAG attention-driven grounding instead of direct bounding-box prediction | MiniCPMV2.5 · OCG average | accuracy: 48.1% -> 84.5% | Helped |
| Use TAG attention-driven grounding instead of direct localization prediction | MiniCPMV2.5 · ScreenSpot average | accuracy: 36.0% -> 54.8% | Helped |
| Use TAG instead of the direct MiniCPMV2.5 grounding method | MiniCPMV2.5 · Mind2Web average | element accuracy: 15.7% -> 24.3% | Helped |
| Use TAG instead of the direct MiniCPMV2.5 method | MiniCPMV2.5 · VisualWebBench Element Grounding | accuracy: 78.4% -> 87.4% | Helped |
| Use TAG instead of MiniCPMV2.5 | MiniCPMV2.5 · ScreenSpot-Desktop | time cost: 1.10 -> 1.04 | Helped |
| Add attention-driven grounding (ablation) | TAG · ScreenSpot Mobile Text | accuracy: 40.3% -> 71.8% | Helped |
| Add adaptive text token selection (ablation) | TAG with attention-driven grounding · ScreenSpot Mobile Text | accuracy: 71.8% -> 86.4% | Helped |
| Add self-attention filtering (ablation) | TAG with attention-driven grounding and token selection · ScreenSpot Mobile Text | accuracy: 86.4% -> 88.3% | Helped |
| Add adaptive text token selection (ablation) | TAG with attention-driven grounding · ScreenSpot Mobile Icon/Widget | accuracy: 27.1% -> 28.4% | Helped |
| Add self-attention filtering (ablation) | TAG with attention-driven grounding and token selection · ScreenSpot Desktop Icon/Widget | accuracy: 24.3% -> 28.6% | Helped |
| Set top-K attention heads to K=10 instead of K=1 or K=1024 (ablation) | TAG · OCG aspect-ratio settings | accuracy | Helped |
| Set threshold to δ=0.5 rather than lower or higher thresholds (ablation) | TAG · GUI grounding datasets | accuracy | Helped |
Surprising findings
- Attention-based grounding without any fine-tuning matches or exceeds SeeClick, a GUI-specific SFT model, on ScreenSpot and OCG.
- MiniCPMV2.5 direct box prediction varies sharply across aspect ratios (80.2% at 4:3 vs 13.6% at 9:21).
Limitations
- Performance depends heavily on the quality and biases of the underlying pretrained MLLM.
- Icon/widget grounding remains low (about 28-29% on ScreenSpot mobile and desktop), so gains are concentrated on text.
- Method relies on prompt design and head selection hyperparameters (K, threshold delta) tuned on the evaluation data.
Experimental setup
Datasets: Mind2Web ScreenSpot VisualWebBench
Benchmarks: OCG ScreenSpot Mind2Web VisualWebBench
Metrics: accuracy (point within ground-truth box) element accuracy
Baselines: MiniCPM-Llama3-V 2.5 SeeClick Qwen-VL-Chat CogAgent MiniGPT-v2 GPT-4V Claude Sonnet
Builds on: MiniCPM-Llama3-V 2.5 SeeClick Set-of-Mark
Closest papers by meaning
- Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion (ICML 2026 · similarity 0.92)
- Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis (NeurIPS 2025 · similarity 0.89)
- Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.89)
- GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding (ICML 2026 · similarity 0.88)
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.88)
- FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.