Research map / Vision-language and multimodal / Robot action and navigation

Code2World: A GUI World Model via Renderable Code Generation

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Embodied multimodal LLM agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Code2World, a Qwen3-VL-8B-based model that predicts the next GUI screen by generating renderable HTML, trained on an 80K-sample HTML corpus (AndroidCode) via SFT then render-aware GRPO with VLM-judge rewards.

Problem

Existing GUI world models are text-based (losing visual detail) or pixel-based (struggling with discrete structure and text-rich layouts), so action-conditioned next-screen prediction lacks both high visual fidelity and structural controllability.

Approach

Treat GUI state transition as renderable code generation: the VLM emits HTML for the next screen, which a browser renders deterministically; training uses SFT on GPT-5-synthesized HTML followed by GRPO with rewards from rendered-image VLM judging and action-consistency judging.

Key results

• Code2World-8B reaches S_ad 94.28 and S_id 88.64 on Android Control, matching or exceeding GPT-5 (94.02 / 90.15 on S_ad/S_id) and beating all open-source baselines up to 106B.
• Adding Code2World to Qwen2.5-VL-7B raises AndroidControl-High grounding from 69.64 to 74.87; adding it to Mobile-Agent-v3-7B raises type from 82.05 to 84.13.
• On AndroidWorld, Gemini-2.5-Flash success rate rises from 41.4% to 50.9% (+9.5) with the full Code2World simulator in the propose-simulate-select pipeline.

Surprising findings

• Using only the visual semantic reward (R_sem) leaves functional logic stagnant or slightly regresses, which the authors attribute to reward hacking toward pixel alignment.
• A naive simulator (Qwen3-VL-8B) adds only +1.2% to Gemini-2.5-Flash success rate, while the SFT model adds +6.1%.

Limitations

• Relies on the VLM judge (Qwen3-VL-8B) for both rewards and metrics, so reward and evaluation may share biases; no human evaluation reported.
• Image assets are replaced by gray text placeholders, so visual fidelity is measured against abstracted rendering rather than true pixels.
• Training data synthesized by GPT-5 from AndroidControl screenshots; the AndroidWorld results are on a single agent setup with one base judge.

Experimental setup

Datasets: AndroidControl GUI Odyssey AndroidCode

Benchmarks: Android Control GUI Odyssey AndroidControl-High AndroidWorld

Metrics: Action Adherence Action Identifiability Element Alignment Layout Integrity SigLIP similarity DINO similarity Action Type accuracy Grounding accuracy success rate

Baselines: Gemini-3-Pro-Image GPT-Image-1 Doubao-Seedream-4.5 Qwen-Image-Edit-Max Janus-Pro-7B GPT-5 Gemini-3-Flash Claude-Sonnet-4.5 JanusCoderV-7B Qwen3-VL-8B Qwen2.5-VL-72B InternVL3-78B

Builds on: GRPO Qwen3-VL AndroidControl Mobile-Agent-v3 AndroidWorld

Closest papers by meaning

• Generative Visual Code Mobile World Models (ICML 2026 · similarity 0.90)
• UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization (ICML 2026 · similarity 0.88)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.88)
• PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control (NeurIPS 2026 · similarity 0.86)
• GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.86)
• GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.86)