Research map / Image generation / Fast diffusion sampling
Decentralized Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an ensemble of independent diffusion expert models on data clusters with no cross-communication, combined at inference by a learned router, and distills them into a dense model.
Problem
Diffusion model training requires synchronized, centralized high-bandwidth GPU clusters, which are costly and inaccessible to many researchers and hard to power at foundation scale.
Approach
Decentralized Flow Matching decomposes the marginal flow over data clusters into a router-weighted sum of data-expert flows, so each expert DiT can be trained fully independently on one cluster, and a separately trained router combines them at test time, matching the monolithic objective.
Key results
- On ImageNet, the 8-expert DDM achieves FID 6.081 vs 8.494 for the monolith at 800k steps (28% lower FID, same training FLOPs).
- On LAION Aesthetics, the DDM reaches FID 6.48 at 200k steps, better than the monolith's 6.52 at 800k steps (about 4x speedup).
- Scaling to eight 3B-parameter MMDiT experts, each trained on a single eight-GPU node for about a week, yields high-resolution samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train an 8-expert decentralized model instead of a monolithic model at matched training FLOPs | DiT XL/2 · ImageNet, 800k steps | FID: 8.494 -> 6.081 | Helped |
| Train an 8-expert decentralized model instead of a monolithic model | DiT XL/2 · LAION Aesthetics | FID: 6.52 -> 6.48 | Helped |
| Distill the 8-expert DDM into a dense model (ablation) | DDM · LAION Aesthetics, 400k steps | FID: 7.82 -> 7.76 | No clear effect |
| Select the top expert rather than combine all experts (ablation) | DDM inference · ImageNet | FID: 10.52 -> 9.84 | Helped |
| Select the top two experts rather than the top expert (ablation) | DDM inference · ImageNet | FID: 9.84 -> 10.31 | Hurt |
| Select the top three experts rather than the top expert (ablation) | DDM inference · ImageNet | FID: 9.84 -> 10.37 | Hurt |
| Sample one expert from the router distribution rather than select the top expert (ablation) | DDM inference · ImageNet | FID: 9.84 -> 157.05 | Hurt |
| Sample two experts from the router distribution rather than select the top expert (ablation) | DDM inference · ImageNet | FID: 9.84 -> 10.27 | Hurt |
| Sample three experts from the router distribution rather than select the top expert (ablation) | DDM inference · ImageNet | FID: 9.84 -> 10.44 | Hurt |
| Use experts above router probability threshold 0.01 rather than select the top expert (ablation) | DDM inference · ImageNet | FID: 9.84 -> 10.46 | Hurt |
| Use experts above router probability threshold 0.05 rather than select the top expert (ablation) | DDM inference · ImageNet | FID: 9.84 -> 10.37 | Hurt |
| Use experts above router probability threshold 0.1 rather than select the top expert (ablation) | DDM inference · ImageNet | FID: 9.84 -> 10.15 | Hurt |
Surprising findings
- Top-1 expert selection at inference outperforms full ensemble weighting and sampling-based combination on ImageNet (FID 9.84 vs 10.52 for full).
Limitations
- Inference uses |K| times more parameters and memory than a monolithic model, motivating distillation.
- Choice of expert count is sensitive; 16 experts train poorly due to small per-expert batch size.
Experimental setup
Datasets: ImageNet LAION-Aesthetics
Benchmarks: ImageNet LAION Aesthetics
Metrics: FID-50K CLIP FID
Baselines: Monolithic DiT (DiT XL/2) DiT B/2 DiT XL/2 Monolith diffusion baseline
Builds on: flow matching DiT Switch Transformers Branch-Train-Merge DiLoCo
Closest papers by meaning
- Accelerating Diffusion Models with Parallel Sampling: Inference at Sub-Linear Time Complexity (NeurIPS 2024 · similarity 0.88)
- DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models (CVPR 2024 · similarity 0.88)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.87)
- Analyzing and Improving the Training Dynamics of Diffusion Models (CVPR 2024 · similarity 0.86)
- AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising (NeurIPS 2024 · similarity 0.86)
- Early-Bird Diffusion: Investigating and Leveraging Timestep-Aware Early-Bird Tickets in Diffusion Models for Efficient Training (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.