Research map / Vision-language and multimodal / Multimodal LLMs

Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds Omni-Diffusion, an any-to-any multimodal model that extends the Dream-7B masked discrete diffusion LLM to jointly model text, speech, and image tokens via a three-stage training pipeline and inference tricks.

Problem

Most multimodal LLMs use autoregressive backbones; the paper asks whether a mask-based discrete diffusion model can serve as a unified backbone for understanding and generating text, speech, and images.

Approach

Model the joint distribution of discretized text, speech, and image tokens with a single mask-token predictor initialized from a pretrained diffusion LLM, with a three-stage progressive training pipeline, attenuated tail-pad masking, and inference techniques (position penalty, special token pre-infilling, adaptive token length) tailored to masked diffusion.

Key results

• On LibriSpeech ASR, Omni-Diffusion reaches 7.05 WER versus AnyGPT's 8.50; on LibriTTS TTS it reaches 3.07 WER versus GLM-4-Voice's 5.64.
• On POPE, MME-P and Seed-2-Plus, it scores 76.6, 1216.7 and 34.5, and it achieves CLIP-T 0.235 and CLIP-I 0.667 on MSCOCO text-to-image.
• On DPG-Bench it scores 68.20 average, above SD v1.5 (63.18) and Show-o (67.48), and below SD v2.0 (68.09).

Surprising findings

• Text-to-image CLIP-T stays nearly unchanged when steps drop from 256 to 10 (0.235 to 0.226), and latency falls from 28.57s to 1.29s.
• Speech-to-image generation at 10 steps is slower (4.25s) than at 50 steps (7.74s) in latency only relative to 256 steps; the latency ordering is not monotone with steps as one might expect from text-to-image.

Limitations

• Image generation quality and VQA results are comparable to, not clearly above, specialized models; DPG-Bench 'Other' score is weak (60.4).
• Limited to text, image, and speech; authors note lack of instruction-based image editing and plan larger scaling.
• Speech-to-image and speech-based evaluations rely on CosyVoice2-synthesized speech and an MME-derived spoken VQA set rather than a standard benchmark.

Experimental setup

Datasets: Tulu 3 SFT mixture Laion-2B LLaVA-OneVision JourneyDB Blip3o-Pretrain-JourneyDB LibriSpeech Common Voice 17 GigaSpeech People's Speech VoxPopuli LibriTTS GLOBE

Benchmarks: LibriSpeech LibriTTS POPE MME-Perception Seed-2-Plus MSCOCO CLIP-T/CLIP-I DPG-Bench

Metrics: WER accuracy CLIP-T CLIP-I latency RTF

Baselines: AnyGPT NExT-GPT CosyVoice GLM-4-Voice Emu DreamLLM LLaVA InstructBLIP mPLUG-Owl

Builds on: Dream-7B MAGVIT-v2 GLM-4-Voice SenseVoiceSmall CosyVoice2

Closest papers by meaning

• Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.92)
• Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.91)
• The Design Space of Tri-Modal Masked Diffusion Models (NeurIPS 2026 · similarity 0.91)
• Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces (ICML 2025 · similarity 0.90)
• Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.90)
• De-Diffusion Makes Text a Strong Cross-Modal Interface (CVPR 2024 · similarity 0.89)