Research map / Vision-language and multimodal / Multimodal LLMs

DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DRS-GUI, a training-free dynamic region search that uses a UI Perceptor (Focus/Shift/Scatter actions) and an MCTS planner with a region quality reward to find the instruction-relevant region before MLLM grounding.

Problem

GUI grounding of instruction-relevant elements in high-resolution, element-dense screenshots; existing single-step and forward-only zoom-in methods cannot recover from early errors.

Approach

Treats grounding as search-then-predict: an MCTS planner schedules three perceptual actions (Focus, Shift, Scatter) over UI-parsed regions, scored by a composite region quality reward (interaction-weighted relevance, UI coverage, semantic concentration), so the base MLLM predicts coordinates only inside the selected region.

Key results

• On ScreenSpot-Pro, DRS-GUI raises Qwen2.5-VL-7B from 26.8% to 40.9% average accuracy (+14.1) and UGround-V1-7B from 31.4% to 45.7%.
• Qwen2.5-VL-3B + DRS-GUI reaches 28.7% on ScreenSpot-Pro, above OS-Atlas-7B (18.9%).
• On ScreenSpot-V2, DRS-GUI lifts UGround-V1-7B from 87.6% to 91.8% and Qwen2.5-VL-7B from 86.5% to 90.5%.

Surprising findings

• A 3B model with DRS-GUI surpasses the 7B OS-Atlas-7B on ScreenSpot-Pro, suggesting search may matter more than model size.

Limitations

• Relies on quality of the UI element parser (OmniParser V2) and text embedder for relevance scores.
• Reward weights and hyperparameters chosen by hand; sensitivity analysis deferred to supplementary material.
• Gains on some subsets (e.g., Mobile on certain actions) are inconsistent; over-shifting can hurt compact layouts.

Experimental setup

Datasets: ScreenSpot-V1 ScreenSpot-V2 ScreenSpot-Pro

Benchmarks: ScreenSpot-V1 ScreenSpot-V2 ScreenSpot-Pro

Metrics: grounding accuracy

Baselines: Qwen2.5-VL-7B UGround-V1-7B UGround-7B OS-Atlas-7B UI-TARS-7B SeeClick CogAgent-18B Aria-UI Claude Computer Use GPT-4o ShowUI-2B

Builds on: Monte Carlo Tree Search UCT OmniParser V2 Instructor-large

Closest papers by meaning

• GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding (ICML 2026 · similarity 0.90)
• Learning GUI Grounding with Spatial Reasoning from Visual Feedback (ICML 2026 · similarity 0.89)
• FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection (CVPR 2026 · similarity 0.89)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.89)
• GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents (NeurIPS 2025 · similarity 0.88)
• Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.88)