Research map / Image generation / Text-to-image diffusion

The Design Space of Tri-Modal Masked Diffusion Models

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Compositional text-to-image guidance · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Pretrains a 3B tri-modal (text, image, audio) masked diffusion model from scratch over a unified discrete token space, and studies its scaling laws, hyperparameter transfer (CompleteP + SDE), and inference/training design choices.

Problem

Multimodal masked diffusion models have mostly been built by finetuning unimodal diffusion LLMs or distilling AR models; the pretraining design space (batch size, hyperparameter transfer, scaling, modality mixing, sampling settings) for native tri-modal MDMs is underexplored.

Approach

Trains a single bidirectional transformer from scratch on text, image-text and audio-text with a unified vocabulary, modality-specific mask/boundary tokens and task tokens; combines CompleteP with an SDE-based batch-size reparameterization to remove the need for batch-size tuning, fits scaling laws in a modified Chinchilla form, and proposes anti-masking as a training variance-reduction trick.

Key results

• Scaling law L(N,D)=E+(A N^{-a/b}+B D^{-1})^b fit with R^2 99.3% and MRE 0.5%, giving D*(N)=7754*N^0.84; a 3B model needs ~480B tokens versus ~60B for Chinchilla AR.
• Anti-masking improves multimodal results at equal compute: FID (Inception) on train data 18.69 -> 17.81 and FAD on LibriSpeech 0.79 -> 0.55.
• Per-module hyperparameter tuning yields a 1.81x reduction in token count to reach equivalent loss at 320-350M scale.

Surprising findings

• Neither the default gamma=0 (more iterations) nor gamma=1 (isohorizon via SDE scaling) drdrift-horizon allocation is optimal; fitted optimum is gamma*=0.44.
• No synergies between modalities at the studied scale: modalities compete for capacity and tokens under mixing.

Limitations

• Scaling laws and ablations are at small scale (320M-1.3B) relative to the 3B flagship model.
• Modality mixing ablations do not show synergies at the tested scale, and conclusions may not transfer to larger scales.

Experimental setup

Datasets: Nemotron-CC DCLM The Pile StackOverflow CC3M CC12M COYO Manzano Whisper-transcribed web audio (2M hours)

Benchmarks: LM evaluation harness GenEval CC12M FID LibriSpeech-PC MMLU ARC-Challenge HellaSwag

Metrics: perplexity/ELBO FID (Inception) FID (DINOv2) GenEval FAD WER Audiobox Aesthetics accuracy

Baselines: LLaDA Dream MMaDA Quokka

Builds on: CompleteP LLaDA MMaDA Chinchilla scaling laws MaskGIT cut-cross-entropy

Closest papers by meaning

• Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion (ICML 2026 · similarity 0.91)
• Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.91)
• Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces (ICML 2025 · similarity 0.90)
• The Diffusion Duality (ICML 2025 · similarity 0.90)
• Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling (ICLR 2025 · similarity 0.89)
• DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation (ICML 2025 · similarity 0.89)