Research map / Vision-language and multimodal / Multimodal LLMs

LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds LLaDA-V, a purely masked-diffusion multimodal LLM that adds a SigLIP2 vision encoder and MLP projector to LLaDA-8B and trains it with a three-stage visual instruction tuning pipeline.

Problem

Whether a purely diffusion-based language model can serve as an effective multimodal LLM for understanding tasks, competitive with autoregressive MLLMs, given that prior diffusion MLLMs had weak language modeling capacity.

Approach

Extend the LLaDA masked diffusion language model to multimodal inputs via visual instruction tuning, using a masked-response training objective over multi-turn dialogues, bidirectional attention, low-confidence remasking at inference, and a multi-stage training curriculum including reasoning data.

Key results

• LLaDA-V reaches 48.6 on MMMU val and 60.1 on MMStar, versus 45.4 and 56.5 for the LLaMA3-V baseline trained on the same data.
• LLaDA-V outperforms LLaMA3-V on 6 of 9 multidisciplinary and math benchmarks and on multi-image/video tasks (MuirBench 48.3 vs 47.4, MLVU 59.5 vs 57.5, VideoMME 56.1 vs 55.8), while lagging on AI2D, DocVQA and RealworldQA.
• LLaDA-V is state-of-the-art among hybrid AR-diffusion and pure diffusion MLLMs in the reported comparisons, but still trails Qwen2-VL on most benchmarks.

Surprising findings

• Despite a weaker language backbone than LLaMA3-8B and Qwen2-7B, LLaDA-V matches or beats the LLaMA3-V baseline on most multidisciplinary and math benchmarks.
• Bidirectional attention across dialogue turns outperformed a dialogue causal mask on most benchmarks, contrary to the expectation that causal masking would reduce train-inference mismatch.

Limitations

• High-resolution image handling by splitting and concatenating SigLIP2 features is less efficient and accurate than native dynamic resolution (e.g., Qwen2-VL).
• Weaker language backbone (LLaDA-8B) leads to lower performance than Qwen2-VL on most benchmarks and weaker chart/document and real-world scene results than LLaMA3-V.
• Possible hallucinations, as with other MLLMs.

Experimental setup

Datasets: LLaVA-Pretrain MAmmoTH-VL single-image (SI-10M) MAmmoTH-VL OneVision (OV-2M) VisualWebInstruct LLaVA-NeXT-Data

Benchmarks: MMMU MMMU-Pro MMStar MME SeedBench MMBench MathVerse MathVista AI2D ChartQA DocVQA InfoVQA

Metrics: accuracy MME perception/cognition score

Baselines: LLaMA3-V Qwen2-VL MetaMorph Show-o JanusFlow Orthus D-DiT LLaVA-OV MAmmoTH

Builds on: LLaDA LLaVA-NeXT visual instruction tuning SigLIP 2 MAmmoTH-VL VisualWebInstruct

Closest papers by meaning

• Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.91)
• Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.91)
• Large Language Diffusion Models (NeurIPS 2025 · similarity 0.91)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training (CVPR 2025 · similarity 0.90)
• Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.90)