Research map / Vision-language and multimodal / Multimodal LLMs

Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trifuse performs training-free GUI grounding by fusing MLLM attention maps with OCR and icon-caption heatmaps via a Consensus-SinglePeak strategy plus two-stage zoom-in localization.

Problem

GUI grounding typically requires costly fine-tuning of MLLMs on large GUI datasets, while existing attention-based training-free methods are unreliable because GUI images lack explicit complementary spatial anchors.

Approach

Filters MLLM attention by token relevance and spatial-entropy head selection, then fuses it with OCR-text and icon-caption heatmaps using a Consensus-SinglePeak fusion (multiplicative consensus plus confidence-weighted single-modality peaks) and a two-stage crop-and-zoom localization.

Key results

• Trifuse with Qwen2.5-VL-3B reaches 81.1 on ScreenSpot and 82.6 on ScreenSpot-v2, versus 57.5 and 51.2 for training-free TAG.
• On ScreenSpot-Pro Trifuse-7B reaches 29.7 versus 3.0 for TAG, and on OSWorld-G 43.6 versus 25.3.
• Applied on top of GUI-AIMA-3B, Trifuse (7B backbone per table) improves ScreenSpot-Pro from 49.8 to 51.3 and ScreenSpot from 88.1 to 90.6.

Surprising findings

• Consensus-only or averaged fusion underperforms single-peak-aware CS fusion; fixed-weight custom fusion drops below attention alone on some benchmarks.
• The method does not handle the refusal (no-target) category of OSWorld-G, limiting gains there.

Limitations

• Performance depends heavily on quality of OCR and icon-caption models, with icon captioning being weak.
• Many hyperparameters make the framework sensitive to settings.
• Performance ceiling set by the pretrained backbone; does not support abstention on OSWorld-G.

Experimental setup

Datasets: ScreenSpot ScreenSpot-v2 ScreenSpot-Pro OSWorld-G

Benchmarks: ScreenSpot ScreenSpot-v2 ScreenSpot-Pro OSWorld-G

Metrics: element accuracy

Baselines: TAG GUI-Actor GUI-AIMA GPT-4o Qwen2.5-VL UGround Aguvis ShowUI UI-TARS JEDI UI-R1 GUI-G1

Builds on: TAG Qwen2.5-VL OmniParser PaddleOCR BGE-M3

Closest papers by meaning

• Attention-Driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models Without Fine-Tuning (AAAI 2025 · similarity 0.92)
• GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding (ICML 2026 · similarity 0.90)
• Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.90)
• FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection (CVPR 2026 · similarity 0.89)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.89)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis (NeurIPS 2025 · similarity 0.89)