Research map / Vision-language and multimodal / Multimodal LLMs

Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalizes Widget-to-Code, releases an image-only widget benchmark with fine-grained visual metrics, and proposes a modular baseline (Perceptual Agent with icon retrieval and templates, WidgetDSL compiler, adaptive rendering) built on Qwen3-VL.

Problem

UI-to-code methods target web and mobile UIs with paired markup, but app widgets are compact, context-free, and lack source code or layout metadata, so faithful image-to-code reconstruction is unexplored.

Approach

Decompose widget images into atomic components with icon retrieval and reusable chart/control templates, generate a framework-agnostic WidgetDSL compiled to React/HTML, and adaptively search container width via browser feedback to satisfy compactness and avoid overflow.

Key results

• Widget2Code (Qwen3-VL-based) reaches Margin 72.15, Text 70.60, Palette 58.09 and Geometry 100.00 versus 64.75/61.17/47.44/95.15 for the Qwen3-VL baseline.
• Specialized UI2Code methods score far lower, e.g., Design2Code Geometry 15.72 and Text 17.50, while GPT-4o and Gemini2.5-Pro are the strongest generalized baselines.
• Human ranking with 25 participants agrees with the metrics, with metric-human correlations of 0.93 (layout), 0.89 (legibility) and 0.94 (style).

Surprising findings

• Specialized UI2Code methods underperform general-purpose MLLMs such as GPT-4o and Gemini2.5-Pro on widgets.
• Fine-tuning only the component extraction module (Ours**) performed best, while full fine-tuning of the 8B model was weaker than the pipeline without fine-tuning in some settings.

Limitations

• Test set is modest (1,000 widgets) and restricted to image-only inputs without source code.
• Fixed set of component templates limits coverage of new or uncovered widget elements.
• Adaptive rendering only rescales the final layout, and the target dimensions are not reproduced exactly without a final resize step.

Experimental setup

Datasets: Widget2Code benchmark images (2,825 curated widgets; 1,000 test) WidgetFactory synthetic data

Benchmarks: Widget2Code benchmark (1,000 test widgets)

Metrics: Margin Content Area Text Jaccard Contrast LocCon Palette Vibrancy Polarity SSIM

Baselines: GPT-4o Gemini2.5-Pro Seed1.6-Thinking Qwen3-VL Qwen3-VL-235b Design2Code DCGen LatCoder UICopilot WebSight-VLM-8B ScreenCoder UI-UG

Builds on: Design2Code SigLIP Qwen3-VL

Closest papers by meaning

• CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design (AAAI 2026 · similarity 0.86)
• Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs (NeurIPS 2024 · similarity 0.85)
• Generative Visual Code Mobile World Models (ICML 2026 · similarity 0.85)
• Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.85)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.85)
• FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection (CVPR 2026 · similarity 0.84)