Research map / Video / Video-language understanding
GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GUI-WORLD, a 12,379-video GUI understanding benchmark and instruction dataset spanning six scenarios, benchmarks MLLMs and video LLMs, and fine-tunes a VideoChat2-based video LLM (GUI-VID) on it.
Problem
Current MLLM GUI agents handle static screenshots in web/mobile settings but struggle with dynamic, sequential GUI content across diverse platforms (desktop, multi-window, XR, iOS).
Approach
A video-based GUI benchmark and instruction-tuning dataset with human-annotated keyframes and GPT-4V-generated captions and QA (static, dynamic, sequential, conversation), plus a progressive two-stage fine-tuned video LLM, GUI-VID, built on VideoChat2.
Key results
- GPT-4o achieves 84.8% average multiple-choice accuracy and 3.573 free-form score across six GUI scenarios, the best among evaluated models.
- GUI-VID improves over its VideoChat2 baseline from 42.9% to 54.6% MCQA accuracy and 2.147 to 2.796 free-form score on average.
- Model-based keyframe selection (UVD+R3M, 3.612) matches human keyframes (3.573) for GPT-4o, while linspace and program-based selection perform worse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune VideoChat2 on GUI-WORLD | VideoChat2 · GUI-WORLD (all scenarios) | MCQA accuracy: 42.9% -> 54.6% | Helped |
| Fine-tune VideoChat2 on GUI-WORLD | VideoChat2 · GUI-WORLD (all scenarios) | Free-form score: 2.147 -> 2.796 | Helped |
| Use UVD+VIP model-based keyframe selection instead of human selection (ablation) | GPT-4o · GUI-WORLD (all scenarios) | Free-form score: 3.573 -> 3.581 | Helped |
| Use UVD+R3M model-based keyframe selection instead of human selection (ablation) | GPT-4o · GUI-WORLD (all scenarios) | Free-form score: 3.573 -> 3.612 | Helped |
| Use programmatic keyframe selection instead of human selection (ablation) | GPT-4o · GUI-WORLD (all scenarios) | Free-form score: 3.573 -> 3.300 | Hurt |
| Use Linspace keyframe selection instead of human selection (ablation) | GPT-4o · GUI-WORLD (all scenarios) | Free-form score: 3.573 -> 3.440 | Hurt |
| Replace visual input with detailed captions (ablation) | GPT-4V · GUI-WORLD (all scenarios) | Free-form score: 3.543 -> 3.316 | Hurt |
| Replace visual input with concise captions (ablation) | GPT-4V · GUI-WORLD (all scenarios) | Free-form score: 3.543 -> 2.971 | Hurt |
| Add vision input alongside detailed captions (ablation) | GPT-4V · GUI-WORLD (all scenarios) | Free-form score: 3.316 -> 3.543 | Helped |
| Remove vision input (ablation) | GPT-4o · GUI-WORLD (all scenarios) | Free-form score: 3.574 -> 2.891 | Hurt |
| Use low-resolution visual input instead of high-resolution input (ablation) | GPT-4o · GUI-WORLD (all scenarios) | Free-form score: 3.574 -> 3.394 | Hurt |
| Add image-text training data alongside video-text data (ablation) | GUI-VID · GUI-WORLD (all scenarios; 8 training frames) | Free-form score: 2.665 -> 2.782 | Helped |
Surprising findings
- Removing MetaGUI and OmniAct data from fine-tuning improved MCQA and XR performance for GUI-VID, while lowering free-form scores in some scenarios.
- Video LLMs lag far behind image LLMs in zero-shot GUI tasks, with ChatUnivi and Minigpt4Video scoring only 22.4% and 16.8% average MCQA accuracy.
Limitations
- Base LLM limits performance; GUI-VID fails at code-generation GUI operation tasks.
- GPT-4V-generated captions and QA were human-verified only on samples, and LLM judges have imperfect human alignment (GPT-4 Pearson 0.856).
Experimental setup
Datasets: GUI-WORLD MetaGUI OmniAct Rico Mind2Web-Multimodal
Benchmarks: GUI-WORLD Mind2Web-Multimodal
Metrics: accuracy (multiple-choice) similarity score 1-5 (LLM-as-a-judge)
Baselines: GPT-4V GPT-4o Gemini-Pro-1.5 Qwen-VL-Max LLaVA-OV-7B ChatUnivi Minigpt4Video VideoChat2 Video-LLaVA
Builds on: VideoChat2 LLaVA UVD Mind2Web MM-Vet
Closest papers by meaning
- MP-GUI: Modality Perception with MLLMs for GUI Understanding (CVPR 2025 · similarity 0.91)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.90)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.90)
- VideoGUI: A Benchmark for GUI Automation from Instructional Videos (NeurIPS 2024 · similarity 0.89)
- Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.89)
- Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.