Research map / Vision-language and multimodal / Multimodal LLMs

UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reformulates UI-to-code as interactive visual optimization and trains UI2Code^N (9B, GLM-4.1V-based) with CPT, SFT, and RL using RVPO, a tournament-based relative preference reward over rendered candidates.

Problem

UI-to-code is usually treated as single-pass generation, which ignores the iterative, render-and-inspect nature of real front-end development and makes absolute visual rewards noisy for RL.

Approach

Relative Visual Policy Optimization (RVPO) derives rewards from win counts in pairwise VLM comparisons among rendered rollouts (tournament aggregation) and feeds them into GRPO without KL regularization, instead of using calibrated absolute scores; UI drafting, polishing, and editing are unified as steps of one closed-loop optimization.

Key results

• UI2Code^N-9B-RL reaches 88.6 on Design2Code and 95.0 on Flame, with 92.5 on Web2Code, versus 79.3 for SFT, per VLM-judge accuracy.
• On UIPolish, UI2Code^N-9B-RL scores 80.0 (Real) and 94.0 (Synthetic), matching or exceeding closed-source models such as Claude-4-5-Sonnet (81.0 Real).
• Test-time refinement raises UI2Code-Real accuracy from 66.0 (N=1) to 74.0 (N=5).

Surprising findings

• CLIP similarity as an RL reward degraded performance relative to SFT on Design2Code and Flame.
• Absolute VLM-based rewards gave only moderate gains and underperformed RVPO.

Limitations

• Relies on a fine-tuned GLM-4.5V verifier whose judgments are only validated on 80-100 human-labeled samples.
• Several benchmarks (UIPolish-bench, UI2Code-Real) are newly constructed and their release is not stated; evaluation largely uses VLM judges with the same or related model families.
• Authors note the work involves a GLM-affiliated judge, creating a potential conflict of interest.

Experimental setup

Datasets: WebCode2M Common Crawl-seeded webpage crawl Mind2Web Design2Code Flame-React-Eval Web2Code

Benchmarks: Design2Code Flame-React-Eval Web2Code UI2Code-Real UIPolish-Real UIPolish-Synthetic Design2Code-HARD UIEdit-Bench

Metrics: VLM-judge accuracy (score >= 80) polishing win accuracy CLIP similarity block/text/position/color match human win/tie/lose absolute positioning ratio

Baselines: InternVL3-9B InternVL3-78B Qwen3-VL-8B-Thinking Qwen3-VL-32B-Instruct Qwen2.5-VL-72B MiMo-VL-7B Kimi-VL-A3B GLM-4.1V-9B-Thinking Claude-4-5-Sonnet-Thinking Claude-4-Sonnet-Thinking Claude-3.7-Sonnet-Thinking GPT-5

Builds on: GRPO GUI-REG GLM-4.1V-9B-Base GLM-4.5V

Closest papers by meaning

• Code2World: A GUI World Model via Renderable Code Generation (NeurIPS 2026 · similarity 0.88)
• FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection (CVPR 2026 · similarity 0.87)
• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning (ICLR 2026 · similarity 0.86)
• JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence (ICLR 2026 · similarity 0.86)
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.85)