Research map / Multi-agent systems and games / LLM agents
Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Vision2Web, a three-level hierarchical benchmark (193 tasks, 1255 test cases, 918 prototypes) for visual website development, evaluated with a workflow-based agent verifier (GUI agent plus VLM judge) across coding agent frameworks.
Problem
Existing coding-agent benchmarks lack end-to-end, visually grounded website development tasks and reliable, reproducible verification of complex interactions and long-horizon outcomes.
Approach
A hierarchical benchmark (static webpage, interactive multi-page frontend, full-stack website) built from real websites, with verification via expert-designed dependency-graph test workflows executed by a GUI agent for functionality and a VLM judge scoring visual fidelity per component.
Key results
- Claude-Opus-4.5 under OpenHands achieves the best full-stack VS/FS of 38.4/57.6, and the best interactive frontend VS/FS of 46.5/66.7.
- Gemini-3-Pro-Preview under OpenHands reaches 55.8 average static webpage score, but drops to VS 11.7 and FS 22.6 on full-stack tasks.
- GUI agent verifier agrees with humans on 218 of 250 nodes (87.2%); VLM judge achieves average Spearman 0.66 vs human inter-annotator 0.78.
Surprising findings
- Qwen3-VL models largely fail to complete the multimodal coding tasks, scoring near zero on interactive and full-stack levels.
- Under most models (excluding Claude), OpenHands yields higher scores than Claude Code, suggesting framework design matters as much as the model.
Limitations
- Evaluation relies on GUI-agent execution and VLM judgments that may introduce residual errors.
- Scope limited to end-to-end website development rather than the full range of software engineering tasks.
- Realistic multimodal verification is costlier than code-only metrics.
Experimental setup
Datasets: C4 validation set
Benchmarks: Vision2Web
Metrics: Visual Score (VS) Functional Score (FS) Deployment Success Rate (DSR) Spearman rank correlation
Baselines: Claude-Opus-4.5 Claude-Sonnet-4.5 GPT-5 Gemini-3-Pro-Preview Gemini-3-Flash-Preview Seed-1.8-VL Qwen3-VL-32B-Instruct Qwen3-VL-8B-Instruct Design2Code WebGenBench SWE-Bench Multimodal
Builds on: WebVoyager Design2Code
Closest papers by meaning
- WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning (ICLR 2026 · similarity 0.90)
- FullStack-Agent: Enhancing Agentic Full-Stack Web Coding via Development-Oriented Testing and Repository Back-Translation (ICML 2026 · similarity 0.88)
- WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.87)
- WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch (NeurIPS 2025 · similarity 0.87)
- AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machines (ICML 2026 · similarity 0.87)
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.