Research map / Image generation / Text-to-image diffusion
Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a framework for multimodal diffusion on arbitrary state spaces with decoupled per-modality time variables, instantiated as continuous-discrete diffusion for text-image generation and mixed-type tabular synthesis.
Problem
Joint generation of multimodal data without relying on modality-unifying tokenizers or VAEs/encoders that harmonize data into a single unimodal latent space, which is problematic for limited data and task-specific designs.
Approach
Builds multimodal diffusion on product state spaces by noising each modality with its own independent time variable (decoupled noise schedules), deriving a unified generalized score-matching objective that reduces to a sum of unimodal denoising losses, enabling unconditional, modality-conditioned, and noisy-guidance generation in one model.
Key results
- On MS-COCO text-to-image, the 481M-parameter model reaches FID 16.16 versus 6.78 for Transfusion (7B) and 26.7 for Chameleon (7B).
- On tabular Trend metric, the ~64K-parameter model is best on Adult (98.75%), Shoppers (98.24%), and News (98.57%), and competitive elsewhere versus TabSyn (~10.7M).
- On MLE, the ~64K model achieves best Beijing RMSE (0.543 vs 0.582 for TabSyn) and best AUC on Magic, Shoppers, and Default ties with TabSyn.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed multimodal diffusion model instead of the strongest listed baseline | Text-to-image generation · MS-COCO | FID: 6.78 -> 16.16 | Hurt |
| Use the proposed native-space model instead of TABSYN | Mixed-type tabular synthesis · Adult | Trend: 98.46 -> 98.75 | Helped |
| Use the proposed native-space model instead of TABSYN | Mixed-type tabular synthesis · Default | Trend: 97.95 -> 96.00 | Hurt |
| Use the proposed native-space model instead of TABSYN | Mixed-type tabular synthesis · Shoppers | Trend: 97.93 -> 98.24 | Helped |
| Use the proposed native-space model instead of TABSYN | Mixed-type tabular synthesis · Magic | Trend: 99.16 -> 98.85 | Hurt |
| Use the proposed native-space model instead of TABSYN | Mixed-type tabular synthesis · Beijing | Trend: 97.76 -> 97.42 | Hurt |
| Use the proposed native-space model instead of TABSYN | Mixed-type tabular synthesis · News | Trend: 98.56 -> 98.57 | Helped |
| Use the proposed native-space model instead of TABSYN | Mixed-type tabular synthesis · Adult | AUC: .915 -> .915 | No clear effect |
| Use the proposed native-space model instead of TABSYN | Mixed-type tabular synthesis · Default | AUC: .764 -> .764 | No clear effect |
| Use the proposed native-space model instead of TABSYN | Mixed-type tabular synthesis · Shoppers | AUC: .920 -> .924 | Helped |
| Use the proposed native-space model instead of TABSYN reproduced | Mixed-type tabular synthesis · Magic | AUC: .939 -> .941 | Helped |
| Use the proposed native-space model instead of TABSYN | Mixed-type tabular synthesis · Beijing | RMSE: .582 -> .543 | Helped |
Surprising findings
- Noisy guidance using a partially noised condition (rather than an unconditional score) improves FID over vanilla classifier-free guidance on MS-COCO.
Limitations
- Did not explore initializing multimodal training from pretrained unimodal diffusion models
- Text-image results are lower than the best industrial models on FID
- Text-image evaluation required caption replication and token truncation to handle distribution shift in caption length
Experimental setup
Datasets: SAM-LLaVA MS-COCO UCI Adult UCI Default UCI Shoppers UCI Magic UCI Beijing UCI News
Benchmarks: MS-COCO FID-30K UCI tabular datasets (Trend, MLE, Shape, Precision, Recall)
Metrics: FID Trend MLE (AUC/RMSE) Shape Precision Recall
Baselines: UniDiffuser Versatile Diffusion UniD3 Show-o Transfusion Chameleon GOGGLE STaSy CoDi TabDDPM TabSyn
Builds on: denoising Markov models MMDiT DiT UniDiffuser Autoguidance
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.92)
- Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion (ICML 2026 · similarity 0.90)
- The Design Space of Tri-Modal Masked Diffusion Models (NeurIPS 2026 · similarity 0.90)
- Diffusion Models for Multi-Task Generative Modeling (ICLR 2024 · similarity 0.89)
- A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (NeurIPS 2024 · similarity 0.89)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.