Research map / Vision-language and multimodal / Multimodal LLMs
From Diagrams to Code: Multilingual Programming with Visual Design
Linzheng Chai, Jian Yang, Shukai Liu, Wei Zhang, Liran WANG, Ke Jin, Tao Sun, Congnan Liu, Chenchen Zhang, Hualei Zhu, Jiaheng Liu, Xianjie Wu and 3 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a visual-first multilingual code generation setting where models must interpret diagrams (architecture/design visuals) and produce code across multiple programming languages.
Problem
AI coding assistants must interpret diagrams of system architectures across multiple programming languages, but such capability is underdeveloped; the abstract is truncated so the specific gap is only partially stated.
Approach
Frames diagram-to-code generation as a visual-first, multilingual programming task motivated by vibe coding, where architectures are communicated through visual designs before coding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add stage-2 SFT to Qwen2-VL-Instruct (ablation) | Qwen2-VL-7B-Instruct · M2Eval | Pass@1 avg: 12.0 -> 16.7 | Improved |
| add stage-2 SFT to Qwen2-VL-Base (ablation) | Qwen2-VL-7B-Base · M2Eval | Pass@1 avg: - -> 18.0 | Improved |
| add stage-1 SFT with vision encoder trained (ablation) | Qwen2-VL-7B-Base · M2Eval | Pass@1 avg: 8.7 (frozen) -> 10.0 | Improved |
| add stage-1 SFT before stage-2 SFT (ablation) | Qwen2-VL-7B-Base · M2Eval | Pass@1 avg: 18.0 -> 25.3 | Improved |
| add stage-2 SFT after stage-1 SFT (ablation) | Qwen2-VL-7B-Base · M2Eval | Pass@1 avg: 10.0 -> 25.3 | Improved |
| use more stage-1 training data before stage-2 (ablation) | M2-Coder-7B · M2Eval | Pass@1 | Improved |
| remove the diagram from the input (ablation) | DeepSeek-V3 and Qwen2.5-Coder · M2Eval | Pass@1: to 0.0 | Hurt |
| train on M2C-Instruct (two-stage) vs. base instruct model | Qwen2-VL-7B · M2Eval | Pass@1 avg: 12.0 -> 25.3 | Improved |
Closest papers by meaning
- JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence (ICLR 2026 · similarity 0.91)
- VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models (ICLR 2026 · similarity 0.90)
- VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing (ICML 2026 · similarity 0.88)
- Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.87)
- Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation (ICLR 2026 · similarity 0.87)
- Can Large Language Models Understand Symbolic Graphics Programs? (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
