Research map / Vision-language and multimodal / Multimodal LLMs

FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FocusUI, an efficient UI grounding framework that selects instruction-relevant visual patches with a lightweight saliency scorer and preserves M-RoPE positional continuity via PosPad (compressing each dropped contiguous run into one marker at its last index).

Problem

UI grounding VLMs process thousands of visual tokens from high-resolution screenshots, causing high compute and attention dilution; general visual token pruning breaks positional continuity and severely hurts precise grounding.

Approach

A query-guided saliency scorer trained with dense per-patch supervision fusing instruction-conditioned bounding-box overlap and a union-find UI-graph prior, plus PosPad, which replaces each contiguous run of dropped visual tokens with one learnable marker at the run's last index to keep (h,w) positions intact.

Key results

• FocusUI-7B (Qwen2.5-VL) reaches 48.3 on ScreenSpot-Pro at full retention, about 3.7 points above GUI-Actor-7B (44.6).
• At 30% visual token retention, FocusUI-7B drops to 45.1 on ScreenSpot-Pro (3.2-point loss) with up to 1.44x faster inference and about 17% lower peak GPU memory.
• Under 30% retention, general pruning methods lose far more accuracy on Qwen2.5-VL-3B (e.g., Vision-Zip 75.4 vs 81.5 on ScreenSpot-V2 average), while FocusUI-3B stays at 91.0 vs 91.5.

Surprising findings

• Naive direct dropping of visual tokens collapses ScreenSpot-Pro accuracy (28.5-29.2 at 50% retention), whereas keeping positional continuity via padding recovers it to about 42.
• Placing the padding marker at the end of each dropped run outperforms placing it at the start or middle.

Limitations

• Efficiency gains come only from spatial visual token reduction; temporal dimension for multi-step UI interaction is not addressed.
• Speedup is modest (up to 1.44x), and peak memory reduction is about 17%.

Experimental setup

Datasets: UGround GUI-Env GUI-Act AndroidControl AMEX Wave-UI

Benchmarks: ScreenSpot-V2 ScreenSpot-Pro OSWorld-G UI-Vision

Metrics: grounding accuracy inference time per sample peak GPU memory Patch Recall@K%

Baselines: GUI-Actor-7B GUI-Actor-3B Jedi-7B Jedi-3B UI-TARS-7B Fast-V HiPrune Vision-Zip

Builds on: GUI-Actor Qwen2.5-VL Qwen3-VL ShowUI DeepStack

Closest papers by meaning

• GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding (ICML 2026 · similarity 0.92)
• Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion (ICML 2026 · similarity 0.89)
• DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding (CVPR 2026 · similarity 0.89)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis (NeurIPS 2025 · similarity 0.89)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.88)
• GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents (NeurIPS 2025 · similarity 0.88)