Research map / Vision-language and multimodal / Multimodal LLMs
Harnessing Webpage UIs for Text-Rich Visual Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthesizes 7.3M multimodal instructions from 1M webpages using text-only LLMs reading accessibility trees, paired with screenshots, to train UIX MLLMs that improve web UI, OCR/doc/chart, and some general tasks.
Problem
Text-rich visual understanding in MLLMs, especially for web UIs, is weak; existing web-derived data (alt-text pairs, simplified HTML, GPT-4 captions) is noisy or narrow.
Approach
Use text-only LLMs (Llama-3-70B-Instruct, GPT-4o-mini for embedded images) to read cleaned accessibility trees with bounding boxes and generate diverse VQA, captioning, OCR, and grounding instructions, then pair them with UI screenshots for two-stage training of MLLMs.
Key results
- UIX-Qwen2 reaches 75.9 on VisualWebBench vs 41.7 for LLaVA-Qwen2 re-implementation (+34.2) and 64.6 for GPT-4V.
- On Mind2Web, UIX-Qwen2-M2W improves element accuracy by up to 19.1 points over LLaVA-Qwen2-M2W (e.g., 43.4 vs 24.3 cross-task).
- UIX-Qwen2 gets DocVQA 85.3 and ChartQA 74.0, outperforming LLaVA-1.6-34B and DocOwl-based baselines on OCR/doc/chart benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with MultiUI + LLaVA-NeXT data | Qwen2-7B-Instruct · VisualWebBench | aggregated score: 41.7 -> 75.9 | Helped |
| Train with MultiUI + LLaVA-NeXT data | Qwen2-7B-Instruct · VisualWebBench Element Ground | accuracy (IoU>0.5): 1.2 -> 66.1 | Helped |
| Train with MultiUI + LLaVA-NeXT data | Qwen2-7B-Instruct · DocVQA | ANLS: 76.5 -> 85.3 | Helped |
| Train with MultiUI + LLaVA-NeXT data | Qwen2-7B-Instruct · OCRBench | accuracy (%): 55.7 -> 66.3 | Helped |
| Train with MultiUI + LLaVA-NeXT data | Qwen2-7B-Instruct · RefCOCO+ | accuracy (IoU>0.5): 75.9 -> 79.1 | Helped |
| Train with MultiUI + LLaVA-NeXT data | Qwen2-7B-Instruct · MMMU | score: 44.7 -> 41.8 | Hurt |
| Train with MultiUI + LLaVA-NeXT data | Qwen2-7B-Instruct · Mind2Web Cross-Task | element accuracy: 24.3 -> 43.4 | Helped |
| Use two-stage rather than single-stage training | UIX · GUI and general scenarios | aggregated performance | Helped |
| Increase MultiUI training data volume | UIX · GUI Grounding | performance: approximately 12% to 60% | Helped |
| Train with QA samples (ablation) | UIX · VisualWebBench | performance | Helped |
| Train with Caption samples (ablation) | UIX · DocVQA | performance | Helped |
| Train with OCR samples (ablation) | UIX · VisualWebBench | performance | Helped |
Surprising findings
- Training on web UI data generalizes to non-UI document, OCR, and chart benchmarks, outperforming specialized document models like DocOwl.
- GPT-4V scores only 64.6 on VisualWebBench, and standard MLLMs nearly fail on GUI grounding (near 0 on element grounding), suggesting web data from LAION/MMC4 does not teach UI skills.
Limitations
- Some general multimodal results are mixed; UIX-Llama3.1 underperforms LLaVA-Llama3.1 on RefCOCO+ (-3.1) and UIX-Qwen2 is lower than LLaVA-Qwen2 on MMMU (41.8 vs 44.7).
- Data generation relies on proprietary or large LLMs (GPT-4o-mini, GPT-4o, Llama-3-70B) and manual prompt tuning.
- Benchmark evaluation uses re-implemented baselines and includes some benchmarks where the authors' models were fine-tuned on training data (M2W).
Experimental setup
Datasets: MultiUI FineWeb CC-MAIN-2024-10 LLaVA-1.5 mix665k LLaVA-NeXT data Mind2Web training set
Benchmarks: VisualWebBench WebSRC ScreenQA-short WidgetCap ScreenSpot RefExp DocVQA ChartQA TextVQA InfoVQA VisualMRC OCRBench
Metrics: aggregated score SQuAD-F1 CIDEr accuracy (IoU>0.5) ANLS relaxed accuracy step success rate element accuracy ROUGE-L
Baselines: LLaVA-1.5 LLaVA-1.6 GPT-4V GPT-4o SeeClick CogAgent ScreenAI Pix2Struct S4 DocOwl-1.5 DocOwl2
Builds on: LLaVA-1.5 LLaVA-NeXT FineWeb Mind2Web
Closest papers by meaning
- Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs (NeurIPS 2024 · similarity 0.89)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.89)
- SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.88)
- Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding (CVPR 2025 · similarity 0.87)
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.87)
- Grounding Multimodal Large Language Models to the World (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.