Research map / Vision-language and multimodal / Multimodal LLMs

CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CANVAS, a 598-task benchmark evaluating VLMs that edit UI designs through tool invocations in Figma via MCP, covering design replication and modification.

Problem

No existing benchmark evaluates vision-language models performing tool-based UI design in design software (e.g., Figma), as prior work focuses on code-based UI generation.

Approach

A multi-turn agentic benchmark where VLMs build or modify Figma designs through context-based tool calls, scored with hierarchical perceptual metrics (SSIM, saliency, BLIP caption similarity, component-wise matching) against ground-truth Figma designs.

Key results

• In design replication, Gemini-2.5-Pro leads SSIM (0.774) and saliency (0.630), while GPT-4.1 leads BLIP (0.655) and component-wise similarity (0.716).
• In design modification, GPT-4.1 achieves the highest scores on all four metrics (e.g., SSIM 0.890, delta +0.183 from base design).
• Logistic regression on human pairwise preferences over 363 non-tie samples reaches 75% accuracy; saliency, BLIP, and component-wise metrics are significant predictors.

Surprising findings

• Gemini-2.5-Pro achieved its highest scores in the single-turn tool-based setting, while GPT-4.1 performed better in multi-turn, suggesting multi-turn agentic iteration is not uniformly beneficial.
• Lower-precision tool use correlated with higher tool diversity, with tool diversity negatively correlated with Pos@K modification scores (rho = -0.365).

Limitations

• Open-source models were excluded due to high failure rates in multi-turn tool invocation.
• Ground-truth designs were cleaned and simplified (placeholders, detached instances, font standardization), reducing fidelity to real-world designs.
• Evaluation uses a limited set of models and a single temperature setting (0).

Experimental setup

Datasets: Figma Community mobile UI designs

Benchmarks: CANVAS

Metrics: SSIM saliency similarity BLIP caption similarity component-wise similarity Pos@K tool precision tool recall tool diversity

Baselines: GPT-4o Claude-3.5-Sonnet Gemini-2.5-Flash Gemini-2.5-Pro code-based single-turn generation

Builds on: ReAct Model Context Protocol Design2Code UEyes BLIP-2

Closest papers by meaning

• UI-Lens: Assessing General MLLMs' Potential to Automate UI Display Quality Assurance (CVPR 2026 · similarity 0.89)
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.89)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.88)
• Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs (CVPR 2026 · similarity 0.86)
• FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection (CVPR 2026 · similarity 0.86)
• Can Vision–Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective. (ICLR 2026 · similarity 0.86)