Research map / Vision-language and multimodal / Multimodal LLMs

VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Merges a coding LLM's task vector into Qwen2.5-VL's language backbone via linear task-vector merging, then SFTs on a new 598k-sample multimodal coding dataset (McD), and introduces InfiBench-V, a visually-rich programming QA benchmark.

Problem

Open-source MLLMs generate code from multimodal inputs (UI screenshots, charts, programming screenshots) poorly, lacking strong coding knowledge while retaining visual understanding.

Approach

Merge the language backbone of a VLM with a code-specialized LLM via linear task-vector arithmetic (vision encoder and projector untouched), so coding skill is added without costly multimodal retraining, followed by SFT on a curated multimodal coding dataset.

Key results

• VisCodex-8B averages 68.8 across four benchmarks, beating GPT-4o-mini (65.7) and all open-source models of 7-15B size.
• VisCodex-33B averages 72.3, close to GPT-4o (73.3) on the same benchmarks.
• Model merging improves VisCodex-8B MMCode pass@1 from 6.8 (w/o merge) to 11.0 and ChartMimic Low-L from 73.4 to 74.8.

Surprising findings

• Direct replacement of the LLM backbone with a code model in a single stage yields a lower Design2Code and ChartMimic score than merging, indicating disrupted visual grounding.
• Code-specialized backbones only gave modest gains on Design2Code (near-saturated), while MMCode gains were larger.

Limitations

• Merge coefficient for 33B model set without tuning due to resource constraints.
• InfiBench-V is small (322 questions) and relies partly on GPT-4o judging.
• Human evaluation covers only 100 questions and five models.

Experimental setup

Datasets: McD Web2Code ChartCoder synthetic Chart2Code KodCode BLIP-558K StackOverflow GitHub matplotlib scripts

Benchmarks: Design2Code ChartMimic MMCode InfiBench-V

Metrics: Low-level score High-level score pass@1 accuracy (keyword matching, unit tests, GPT-4o judge) Block-Match Text Position Color CLIP

Baselines: GPT-4o GPT-4o-mini Qwen2.5-VL-7B-Instruct Qwen2.5-VL-32B-Instruct Qwen2.5-VL-72B-Instruct InternVL3-8B InternVL3-14B InternVL3-78B llava-onevision-qwen2-72b MiniCPM-V-2_6 Llama-3.2-11B-Vision-Instruct

Builds on: task vectors linear model merging Qwen2.5-VL LLaVA-OneVision Web2Code ChartCoder InfiBench

Closest papers by meaning

• JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence (ICLR 2026 · similarity 0.91)
• Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs (NeurIPS 2024 · similarity 0.91)
• Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.90)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.88)
• CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval (CVPR 2026 · similarity 0.88)
• LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.88)