Research map / Vision-language and multimodal / Multimodal LLMs
Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Web2Code, a ~1.18M-example webpage-to-HTML instruction tuning dataset plus an evaluation suite (WUB yes/no QA and WCGB GPT-4V-judged render comparison), showing gains on webpage tasks and general MLLM benchmarks when training LLaVA-style models on multiple LLM backbones.
Problem
Existing MLLMs are poor at understanding webpage screenshots and generating corresponding HTML code, and existing webpage-to-code datasets lack instruction format and do not suit general-purpose MLLM training; text similarity metrics also fail to assess code fidelity.
Approach
Build an instruction-formatted webpage-to-code dataset by generating new HTML pages with GPT-3.5, refining Pix2Code and WebSight and WebSRC with GPT-4, adding GPT-4-generated webpage QA pairs, and evaluate code by rendering predicted HTML to screenshots and scoring them with GPT-4V on 10 criteria.
Key results
- LLaMA3-8B with full Web2Code data reaches WCGB overall 8.530 (from 1.79 with general data only) and WUB accuracy 74.84% (from 65.56%).
- Web2Code-trained CrystalChat-7B adds gains on POPE (86.86 to 87.10) and TextVQA (57.84 to 58.15) while keeping general benchmarks comparable.
- On WebSight comparison, 60K DWCG samples outperform the 823K WebSight v0.1 data on WCGB overall (7.990 vs 7.701) with CrystalChat-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DWCG to general-domain instruction tuning (ablation) | LLaMA3-8B · WCGB | Overall: 1.79→6.402 | Helped |
| Add DWU to DWCG training (ablation) | LLaMA3-8B · WCGB | Overall: 6.402→6.716 | Helped |
| Add DWCG_R to DWCG + DWU training (ablation) | LLaMA3-8B · WCGB | Overall: 6.716→7.806 | Helped |
| Add DWU_R to DWCG + DWU + DWCG_R training (ablation) | LLaMA3-8B · WCGB | Overall: 7.806→8.530 | Helped |
| Add DWCG to general-domain instruction tuning (ablation) | LLaMA3-8B · WUB | Accuracy (%): 65.56→60.00 | Hurt |
| Add DWU to DWCG training (ablation) | LLaMA3-8B · WUB | Accuracy (%): 60.00→69.33 | Helped |
| Add DWCG_R to DWCG + DWU training (ablation) | LLaMA3-8B · WUB | Accuracy (%): 69.33→68.68 | Hurt |
| Add all Web2Code components to general-domain instruction tuning (ablation) | LLaMA3-8B · WUB | Accuracy (%): 65.56→74.84 | Helped |
| Add DWU to DWCG training (ablation) | CrystalCoder-7B · WUB | Accuracy (%): 71.81→73.74 | Helped |
| Use Web2Code instead of WebSight v0.1 | LLaMA3-8B · WCGB | Overall: 4.125→8.530 | Helped |
| Use Web2Code instead of WebSight v0.2 | LLaMA3-8B · WUB | Accuracy (%): 57.33→74.84 | Helped |
| Add DWU to CrystalChat-7B training (ablation) | CrystalChat-7B with general-domain data · POPE | score: 86.86→87.10 | Helped |
Surprising findings
- Training on the WebSight dataset alone substantially degrades WUB accuracy (e.g., 59.38% vs 73.54%) relative to general-domain-only training on CrystalChat-7B.
- Adding DWCG data improved MME-P by about 40 points for CrystalChat-7B despite being code-focused.
Limitations
- Dataset may be biased and not cover all HTML coding scenarios.
- Some webpages contain privacy-sensitive content; human check found 2.7% of sampled data with personal info.
- Evaluation via GPT-4V scoring may not capture code efficiency, readability, or best practices.
Experimental setup
Datasets: Web2Code (DWCG, DWU, DWCG_R, DWU_R) WebSight Pix2Code WebSRC LLaVA instruction data
Benchmarks: WUB WCGB Design2Code metrics MME POPE ScienceQA TextVQA
Metrics: accuracy GPT-4V 0-10 criteria score block-match text position color CLIP score
Baselines: LLaVA-1.5 WebSight v0.1 WebSight v0.2 Vicuna1.5-7B CrystalCoder-7B CodeLlama-7B LLaMA2-7B
Builds on: LLaVA-1.5 CodeAlpaca Design2Code WebSRC Pix2Code
Closest papers by meaning
- VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models (ICLR 2026 · similarity 0.91)
- Harnessing Webpage UIs for Text-Rich Visual Understanding (ICLR 2025 · similarity 0.89)
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (NeurIPS 2024 · similarity 0.89)
- MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? (ICLR 2025 · similarity 0.88)
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.88)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.