Research map / Vision-language and multimodal / Multimodal LLMs
MP-GUI: Modality Perception with MLLMs for GUI Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MP-GUI, an 8B MLLM for GUI understanding that adds three modality perceivers (text, graphics, spatial) and a task-conditioned fusion gate on top of InternVL2-8B, trained with a four-step curriculum on ~0.68M GUI samples including a new Spatial Relationship Prediction task and synthetic QA data.
Problem
Existing MLLMs treat GUI screenshots like natural images and lack explicit spatial structure modeling, while obtaining reliable screen structure data (view hierarchies, HTML) is hard due to noise and privacy issues.
Approach
Add three GUI-tailored perceivers (textual, graphical, spatial) over the vision backbone, fuse their signals with a question-conditioned Fusion Gate to produce task-oriented visual clues, train each perceiver stage-wise with modality-specific data (including a novel SRP task from VH trees), and use MLLM-synthesized QA data to train the fusion gate.
Key results
- MP-GUI (8B) reaches 151.0 CIDEr on Widget Captioning and 83.0 Acc@IoU=0.1 on RefExp, best among compared MLLMs in the basic GUI benchmark.
- On AITW navigation, MP-GUI achieves 69.2 overall vs. 68.3 for ShowUI and 62.0 for InternVL2.
- Using 0.68M GUI-specific samples, MP-GUI is competitive with or beats GUI-specific methods trained on far more data (e.g., ScreenAI with 383.5M samples) on most tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MP-GUI's GUI-tailored perceivers, fusion gate, and multi-stage training rather than using InternVL2-P | InternVL2-8B · Widget Captioning | CIDEr: 142.8 -> 151.0 | Helped |
| Add MP-GUI's GUI-tailored perceivers, fusion gate, and multi-stage training rather than using InternVL2-P | InternVL2-8B · RefExp | Acc@IoU=0.1: 72.4 -> 83.0 | Helped |
| Add MP-GUI's GUI-tailored perceivers, fusion gate, and multi-stage training rather than using InternVL2-P | InternVL2-8B · ScreenQA Short | SQuAD F1: 88.9 -> 90.5 | Hurt |
| Add MP-GUI's GUI-tailored perceivers, fusion gate, and multi-stage training rather than using InternVL2-P | InternVL2-8B · Taperception | F1: 87.8 -> 88.2 | Hurt |
| Add task-oriented FG pretraining in Step 4 (ablation) | MP-GUI · Widget Captioning | CIDEr: 142.1 -> 151.0 | Helped |
| Add task-oriented FG pretraining in Step 4 (ablation) | MP-GUI · Complex ScreenQA | SQuAD F1: 80.7 -> 84.3 | Helped |
| Use the Fusion Gate instead of averaging the three perceiver outputs (ablation) | MP-GUI · Widget Captioning | CIDEr: 143.4 -> 151.0 | Helped |
| Add the Textual Perceiver (ablation) | MP-GUI · Screen2Words | CIDEr: 115.2 -> 118.4 | Helped |
| Add the Graphical Perceiver (ablation) | MP-GUI · Complex ScreenQA | SQuAD F1: 80.3 -> 84.3 | Helped |
| Add the Spatial Perceiver (ablation) | MP-GUI · Widget Captioning | CIDEr: 141.9 -> 151.0 | Helped |
| Train TGS-PFM step by step with MTS instead of end to end (ablation) | MP-GUI · WebSRC | SQuAD F1: 86.9 -> 89.2 | Helped |
Surprising findings
- Instruction-tuning InternVL2 on domain GUI data (InternVL2-P) hurts reasoning-heavy CQA and QAS relative to the base InternVL2.
- Training on mobile-only screenshots transfers to desktop and website grounding in zero-shot settings.
Limitations
- Synthetic QA labels come from Qwen2-VL-72B, which the authors note is not on par with human supervision.
- Relies on a YOLOv8 detector fine-tuned on about 9k in-house labeled Alipay screens for local description data, which is not public.
- Some gains are marginal (e.g., Taperception and WebSRC are second-best or within 1 point of baselines).
Experimental setup
Datasets: AMEX AITW Semantic UI ScreenQA WebSRC RefExp Widget Captioning Taperception Screen2Words Mind2Web ScreenSpot
Benchmarks: Widget Captioning Screen2Words RefExp Taperception WebSRC ScreenQA ScreenQA Short Complex ScreenQA AITW Mind2Web ScreenSpot
Metrics: CIDEr F1 ROUGE-L SQuAD F1 Acc@IoU Step.SR Ele.Acc Op.F1
Baselines: InternVL2 InternVL2-P Qwen2-VL Qwen-VL MiniCPM-V 2.6 Llama 3.2-V CogAgent SeeClick ShowUI ScreenAI
Builds on: InternVL2 Mixture of Experts LoRA Set-of-Marks
Closest papers by meaning
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.91)
- Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models (CVPR 2026 · similarity 0.86)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.86)
- Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.86)
- Attention-Driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models Without Fine-Tuning (AAAI 2025 · similarity 0.86)
- DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.