Research map / Vision-language and multimodal / Multimodal LLMs
Figma2Code: Automating Multimodal Design to Code in the Wild
Yi Gui, Jiawan Zhang, Yina Wang, Tianran Ma, Yao Wan, Shilin He, Dongping Chen, Zhou Zhao, Wenbin Jiang, Xuanhua Shi, Hai Jin, Philip S. Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Figma2Code, a multimodal design-to-code task and 213-sample benchmark built from Figma community files (image + JSON metadata + assets), benchmarking ten MLLMs and an agentic workflow on visual fidelity, responsiveness, and maintainability.
Problem
Image-only design-to-code forces MLLMs to infer UI details such as icons, hierarchy, styles, and layout from screenshots, and ignores the metadata and assets present in real Figma design files. Existing benchmarks lack this multimodal information and code-quality evaluation.
Approach
Pairs Figma design images with cleaned JSON metadata and assets collected from the Figma community, and evaluates generated Tailwind code on visual fidelity (VES, MAE) plus reference-free responsiveness (RUR, APR) and maintainability (STR, AVU) metrics, with an F2CAgent ReAct-style critic/refiner loop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Figma metadata to image input | Direct Prompting ERNIE 4.5 424B VL · Figma2Code | VES: 0.5653 -> 0.6801 | Improved |
| add design image to metadata input | Direct Prompting ERNIE 4.5 424B VL · Figma2Code | VES: 0.6801 -> 0.6923 | Improved |
| add F2CAgent workflow on image + metadata | ERNIE 4.5 424B VL · Figma2Code | VES: 0.6923 -> 0.7990 | Improved |
| add F2CAgent workflow on image + metadata | ERNIE 4.5 424B VL · Figma2Code | APR (%): 2.83 -> 13.57 | Hurt |
| add metadata to image input | Direct Prompting ERNIE 4.5 424B VL · Figma2Code | APR (%): 0.01 -> 5.01 | Hurt |
| use Template Conversion instead of MLLM prompting on metadata | FigmaToCode plugin · Figma2Code | APR (%): 5.01 -> 58.0 | Hurt |
| add text-augmentation to image-only prompting | Design2Code ERNIE 4.5 424B VL · Figma2Code | VES: 0.5653 -> 0.5683 | No clear effect |
| add style attributes to metadata (ablation) | Direct Prompting ERNIE 4.5 424B VL · Figma2Code | VES: below 0.55 -> approximately 0.65 | Improved |
| add image assets to metadata (ablation) | Direct Prompting ERNIE 4.5 424B VL · Figma2Code | VES | Improved |
| add geometry and hierarchy to metadata (ablation) | Direct Prompting ERNIE 4.5 424B VL · Figma2Code | RUR | Improved |
| add style or geometry to metadata (ablation) | Direct Prompting ERNIE 4.5 424B VL · Figma2Code | STR / AVU | Hurt |
| add image to metadata-only input in ablations | Direct Prompting ERNIE 4.5 424B VL · Figma2Code | VES / APR | Improved |
Surprising findings
- Template conversion from metadata achieves competitive MAE (0.1205) but collapses on responsiveness (RUR 0%) and maintainability (STR 0.01%).
- Stronger proprietary models achieve higher visual fidelity yet produce less responsive and less maintainable code than some open-source models.
Limitations
- Benchmark is small (213 test samples) and targets Tailwind CSS only.
- Metrics are reference-free and heuristic; the agentic F2CAgent is preliminary and only tested with one backbone.
- Dataset restricted to free CC BY community files and non-commercial use.
Experimental setup
Datasets: Figma community design files (crawled)
Benchmarks: Figma2Code
Metrics: VES MAE RUR APR STR AVU
Baselines: Design2Code direct prompting Text-Augmented prompting FigmaToCode template conversion Llama 4 Maverick Llama 4 Scout Qwen 2.5 VL ERNIE 4.5 424B VL Nova Pro v1 Gemini 2.5 Pro Grok4 Claude Opus 4.1 GPT-4o
Builds on: Design2Code ReAct FigmaToCode DINOv2
Closest papers by meaning
- Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs (CVPR 2026 · similarity 0.88)
- JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence (ICLR 2026 · similarity 0.87)
- CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design (AAAI 2026 · similarity 0.87)
- VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models (ICLR 2026 · similarity 0.86)
- Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs (NeurIPS 2024 · similarity 0.85)
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
