Research map / Vision-language and multimodal / Multimodal LLMs

MMaDA: Multimodal Large Diffusion Language Models

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 16 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMaDA, an 8B unified multimodal diffusion language model trained with a shared masked-token objective, mixed long-CoT finetuning, and a diffusion GRPO variant (UniGRPO) for reasoning, understanding, and text-to-image generation.

Problem

Existing unified multimodal foundation models focus on architecture and pretraining, leaving post-training (CoT finetuning and RL) largely unexplored, especially for non-autoregressive diffusion models.

Approach

A modality-agnostic masked-token diffusion model over discrete text and image tokens, with a unified task-agnostic CoT format for cold-start finetuning and a UniGRPO policy-gradient algorithm that handles diffusion log-likelihoods via random answer masking and a unified reward mix (correctness, format, CLIP, ImageReward).

Key results

• MMaDA-8B reaches 73.4 on GSM8K and 36.0 on MATH500 after UniGRPO, versus 70.7 and 27.3 for LLaDA-8B.
• On GenEval overall, MMaDA scores 0.63, above Janus (0.61) and Show-o (0.53); its CLIP score is 32.46 and ImageReward 1.15.
• On multimodal understanding, MMaDA reports POPE 86.1, VQAv2 76.7, and MMB 68.5, outperforming the listed unified models on several benchmarks.

Measured effects reported in the paper

Surprising findings

• MMaDA trails Qwen2-7B on MMLU-style and math benchmarks (e.g., GSM8K 73.4 vs 80.2), and underperforms LLaMA-3-8B-level baselines on TruthfulQA (43.1 vs 44.0).

Limitations

• Model size is limited to 8B parameters, which the authors acknowledge restricts performance.
• Text-only language evaluation relies on limited task-specific tokens and open-source text data.
• Some comparisons use reported numbers from other papers and mix evaluation protocols.

Experimental setup

Datasets: RefinedWeb ImageNet-1k Alpaca LLaVA-1.5 ReasonFlux LIMO s1k OpenThoughts AceMath-Instruct GeoQA CLEVR GSM8K

Benchmarks: POPE MME Flickr30k VQAv2 GQA MMMU MMBench SEED GenEval WISE CLIP Score ImageReward

Metrics: accuracy CLIP score ImageReward GenEval overall

Baselines: Show-o Janus Emu3 LLaDA-8B LLaMA-2-7B Qwen2-7B SEED-X DreamLLM

Builds on: LLaDA Show-o MAGVIT-v2 GRPO d1

Closest papers by meaning

• I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.91)
• Lavida-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models (ICML 2026 · similarity 0.91)
• Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.91)
• Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.91)
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
• Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Models (NeurIPS 2026 · similarity 0.89)