Research map / Vision-language and multimodal / Multimodal LLMs

Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts the discrete diffusion VLM LLaDA-V to single-turn GUI grounding and proposes a hybrid linear/deterministic masking schedule that improves step success rate over linear masking, while comparing against AR VLMs.

Problem

Whether discrete diffusion vision-language models (DVLMs) can serve as a viable alternative to autoregressive VLMs for GUI grounding, which requires predicting action types and bounding boxes from screenshots and instructions.

Approach

Adapts LLaDA-V to output action strings and introduces a hybrid masking schedule: a linear-masking phase predicts action type and anchor (x1,y1), then a fully deterministic masking phase predicts the box extent (x2,y2) conditioned on the anchor.

Key results

• Hybrid masking improves SSR over linear masking by +1.6 (Mind2Web), +5.3 (ScreenSpot-Web), +1.3 (ScreenSpot-Text), and +6.1 (VisualWebArena) points.
• LLaDA-V 8B with hybrid masking reaches 83.9% SSR on Mind2Web, close to Qwen2.5-VL 7B at 81.9%, but lower on ScreenSpot-Icon (63.1 vs 85.4).
• Scaling to 120K multi-domain samples reduces latency by roughly 1.2-1.4 seconds and improves SSR by an average of 17-20 points across benchmarks.

Surprising findings

• Zero-shot LLaDA-V achieves near-zero SSR on Mind2Web, while the 7K-sample fine-tuned model generalizes to unseen ScreenSpot and VisualWebArena benchmarks.
• Hybrid masking increases latency (e.g., 3.02s to 5.44s on Mind2Web) even though it improves accuracy.

Limitations

• Still lags AR models in latency and accuracy, particularly on ScreenSpot-Web-Icon and SWT.
• Restricted to single-step action prediction; multi-step planning not studied.
• Hybrid masking adds sequential computation and latency.

Experimental setup

Datasets: Mind2Web WebLinX OS-Atlas Rico Widget Caption ScreenSpot-Web-Text ScreenSpot-Web-Icon VisualWebArena

Benchmarks: Mind2Web ScreenSpot-Web-Text ScreenSpot-Web-Icon VisualWebArena

Metrics: Step Success Rate (SSR) Action-Type F1 latency converged steps

Baselines: Qwen2.5-VL (3B) Qwen2.5-VL (7B) Phi-3-Vision LLaDA-V 8B (linear masking)

Builds on: LLaDA-V LLaDA SigLIP-2 LLaVA-NeXT

Closest papers by meaning

• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.92)
• GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents (NeurIPS 2025 · similarity 0.90)
• Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.90)
• GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding (ICML 2026 · similarity 0.90)
• Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion (ICML 2026 · similarity 0.90)
• GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning (ICLR 2026 · similarity 0.89)