Research map / Large language models / Agents, code and math
WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WebGen-Bench, a benchmark of 101 instructions and 647 test cases for LLM agents building multi-file websites from scratch, scored by a web-navigation UI agent, plus WebGen-Instruct training set and WebGen-LM fine-tuned Qwen2.5-Coder models.
Problem
Evaluating LLM-based code agents on generating complete, functional, multi-file websites from natural-language instructions, which existing repository-level benchmarks (bug fixing, patching) do not cover.
Approach
Build from-scratch website-generation instructions across 20 application categories, generate per-requirement test cases with GPT-4o (manually validated), and automatically grade websites with a UI web-navigation agent (WebVoyager with Qwen2.5-VL-32B) plus a GPT-4o appearance score; fine-tune Qwen2.5-Coder on rejection-sampled Bolt.diy trajectories.
Key results
- Bolt.diy with DeepSeek-R1 achieves the best general-model accuracy of 27.8% on WebGen-Bench's 647 test cases.
- WebGen-LM-32B (Qwen2.5-Coder-32B fine-tuned on 600 trajectories) reaches 38.2% accuracy, surpassing DeepSeek-R1.
- UI agent results align with manual testing on 86.1-94.4% of test cases for three models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-Coder-32B-Instruct on 600 Bolt.diy trajectories | Qwen2.5-Coder-32B-Instruct · WebGen-Bench | accuracy: 9.5 -> 38.2 | Helped |
| Increase the fine-tuning set from 150 to 300 to 600 samples (ablation) | Qwen2.5-Coder-32B-Instruct · WebGen-Bench | accuracy | Helped |
| Use Bolt.diy instead of OpenHands with DeepSeek-R1 | DeepSeek-R1-powered code agents · WebGen-Bench | accuracy: 10.2 -> 27.8 | Helped |
| Use Bolt.diy instead of Aider with DeepSeek-R1 | DeepSeek-R1-powered code agents · WebGen-Bench | accuracy: 27.7 -> 27.8 | No clear effect |
Surprising findings
- Fine-tuning on only 600 trajectories lifted Qwen2.5-Coder-32B from 9.5% to 38.2%, above the best proprietary model.
- Smaller general open-source models such as Qwen2.5-Coder-32B and Qwen2.5-72B show large gaps versus proprietary models, and Claude-3.5-Sonnet has the highest appearance score despite lower accuracy than DeepSeek-R1.
Limitations
- Websites are primarily built with TypeScript, JavaScript, CSS and HTML; other languages like Python or Java are not covered.
- Only supervised fine-tuning was used; RL or DPO were not explored.
- Test set is small (101 instructions, 647 test cases), and the training set was limited to 600 sampled trajectories due to API budget.
Experimental setup
Datasets: WebGen-Instruct WebGen-Bench
Benchmarks: WebGen-Bench
Metrics: accuracy (YES=1, PARTIAL=0.5) appearance score (1-5) alignment rate with human testers
Baselines: Claude-3.5-Sonnet DeepSeek-R1 DeepSeek-V3 GPT-4o o3-mini Qwen2.5-Coder-32B Qwen2.5-72B-Instruct
Builds on: WebVoyager Bolt.diy OpenHands Aider SWE-bench
Closest papers by meaning
- WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning (ICLR 2026 · similarity 0.90)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.89)
- FullStack-Agent: Enhancing Agentic Full-Stack Web Coding via Development-Oriented Testing and Repository Back-Translation (ICML 2026 · similarity 0.89)
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (ICLR 2024 · similarity 0.88)
- How to Train Your LLM Web Agent: A Statistical Diagnosis (NeurIPS 2025 · similarity 0.88)
- Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.