Research map / Image generation / Fast diffusion sampling
How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives the first theoretical generation error bound (KL divergence) for distributed diffusion models trained by workers with resource-constrained sparse/pruned local training and coordinate-wise aggregation, linear in data dimension d.
Problem
Single-worker theory of diffusion models does not cover distributed training where workers have heterogeneous resources (forcing sparse/pruned local training) and heterogeneous data, so generation quality of such collaboratively trained models is not theoretically characterized.
Approach
Combines denoising score matching, OU-process SDE discretization analyzed with Girsanov's theorem, and federated-style local training with per-worker binary masks and coordinate-wise averaging on the server; derives a KL generation bound and shows hyperparameter choices (time step, early stopping T, learning rate) make the bound governed by distributed training dynamics.
Key results
- Establishes the first generation error bound for distributed diffusion models under resource limits, linear in data dimension d, matching the single-worker best known results (Benton et al., 2024).
- Remark 4.12: with T=1/2 log((d+M)/F(θ0)) and K=Θ((d+M)(T+log(1/δ))^2/F(θ0)), the KL bound reduces to ∝ F(θ0)+C1 when learning rate and pruning error are controlled.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select noise-schedule and training hyperparameters as specified in Remark 4.12 | distributed diffusion model with local sparse training · worker-n data distribution qn,δ | KL divergence | Helped |
| use local sparse training with coordinate-wise model aggregation | distributed diffusion model · worker-n data distribution qn,δ | KL divergence | Mixed |
| make all workers’ initial data distributions identical | distributed diffusion model · worker-n data distribution qn,δ | KL divergence | Helped |
Limitations
- No empirical experiments validating the bound on real data or models.
- Bound depends on assumptions such as Lipschitz gradients, bounded pruning error and bounded variance, which may be strong.
- Assumes unbiased local stochastic losses and that all workers share the same model architecture for coordinate-wise aggregation.
Experimental setup
Metrics: KL divergence (theoretical bound)
Builds on: Benton et al. 2024 (linear-dimension diffusion error bounds) Chen et al. 2023 (polynomial diffusion error bounds) Lian et al. 2017 (decentralized parallel SGD) Denoising score matching (Vincent 2011) DDPM
Closest papers by meaning
- The Emergence of Reproducibility and Consistency in Diffusion Models (ICML 2024 · similarity 0.90)
- SyncSDE: A Probabilistic Framework for Diffusion Synchronization (CVPR 2025 · similarity 0.88)
- Elucidating the Exposure Bias in Diffusion Models (ICLR 2024 · similarity 0.87)
- Understanding diffusion models requires rethinking (again) generalization (NeurIPS 2026 · similarity 0.86)
- Evaluating the design space of diffusion-based generative models (NeurIPS 2024 · similarity 0.86)
- Constrained Diffusion Models via Dual Training (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.