Research map / Image generation / Autoregressive and latent generation
Diffusion Models for Multi-Task Generative Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MT-Diffusion, a multi-modal diffusion model that aggregates multiple task-data sources in a shared diffusion space and trains a shared U-Net with modality-specific decoder heads under a multi-task ELBO-derived loss.
Problem
Standard diffusion models handle only a single generation task or modality, and there is no principled way to jointly generate heterogeneous modalities (e.g., images and labels) within the diffusion framework.
Approach
Define the forward diffusion process to aggregate information from multiple modality encoders into the mean of the transition, and the reverse process as a shared U-Net with modality-specific decoder heads, trained with a multi-task loss derived from a new multi-modal variational lower bound that generalizes DDPM.
Key results
- On ImageNet-128, MT-Diffusion-X (continued finetuned variant, MT-Diffusion^f) reaches FID 7.01 and IS 84.22 versus ADM 9.62 FID / 74.04 IS under matched continued training, and the masked-image model MT-Diffusion-X reaches FID 9.85 vs ADM unconditional 23.22 at 32/64 resolution comparisons.
- On Cityscapes translation, MT-Diffusion obtains per-pixel accuracy 0.95 and class IOU 0.70 versus pix2pix 0.85 and 0.32, and single-task diffusion 0.72 and 0.32.
- LPIPS for masked-image restoration with 5 masked patches is 0.035 for MT-Diffusion vs 0.400 for SDEdit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with masked-image recovery as an additional task | MT-Diffusion-X · ImageNet-64 | IS: 15.64 -> 34.53 | Helped |
| Condition noise prediction on modality data instead of using only the noisy latent and timestep (ablation) | MT-Diffusion masked-image training · ImageNet-64 | IS: 23.77 -> 34.53 | Helped |
| Use masked-image training with constrained restoration | MT-Diffusion-X · ImageNet masked-image recovery, Mask-5 through Mask-20 | LPIPS: Mask-5: 0.400 -> 0.035; Mask-10: 0.466 -> 0.068; Mask-15: 0.497 -> 0.099; Mask-20: 0.513 -> 0.133 | Helped |
| Finetune the generator on single-task image generation after multi-task training (ablation) | MT-Diffusion · ImageNet-128 | FID: 8.42 -> 7.01 | Helped |
| Finetune the generator on single-task image generation after multi-task training, with classifier guidance (ablation) | MT-Diffusion_g · ImageNet-128 | FID: 8.06 -> 3.51 | Helped |
| Jointly model image and segmentation labels | MT-Diffusion · Cityscape | per-pixel accuracy: 0.72 -> 0.95 | Helped |
| Jointly generate images and labels with the middle-block MLP decoder rather than the output-block classifier decoder (ablation) | MT-Diffusion-M vs MT-Diffusion-E · ImageNet-64 | IS: 11.86 -> 15.66 | Helped |
| Finetune the joint image-label model on single-task generation (ablation) | MT-Diffusion-M · ImageNet-64 | FID: 33.92 -> 13.48 | Helped |
Surprising findings
- Joint image-label training of the generator (MT-Diffusion-M/E) trades off single-task generation quality, with MT-Diffusion-E slightly underperforming single-task ADM before single-task finetuning.
Limitations
- Only two modalities considered; more modalities left to future work.
- Heterogeneous tasks (image-label) can degrade single-task generation quality without extra finetuning.
- Large-scale image-representation experiments are only qualitative with no quantitative comparison.
Experimental setup
Datasets: ImageNet-1K Cityscapes night2day LAION CIFAR-10
Benchmarks: ImageNet-64 ImageNet-128 Cityscapes night2day
Metrics: IS FID sFID Precision Recall LPIPS per-pixel accuracy per-class accuracy class IOU top-1 accuracy
Baselines: ADM SDEdit CycleGAN pix2pix InternImage-H Versatile Diffusion
Builds on: DDPM classifier-free guidance guided diffusion (ADM) Latent Diffusion Stable Diffusion
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.92)
- DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data (CVPR 2024 · similarity 0.90)
- Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces (ICML 2025 · similarity 0.89)
- Denoising Task Difficulty-based Curriculum for Training Diffusion Models (ICLR 2025 · similarity 0.89)
- Neural Diffusion Models (ICML 2024 · similarity 0.89)
- Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.