Research map / Image generation / Fast diffusion sampling
Evaluating the design space of diffusion-based generative models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a non-asymptotic full error analysis of diffusion model generation (denoising score matching trained by GD on a deep ReLU net, plus VESDE sampling error), and uses it to theoretically rationalize EDM's bell-shaped loss weighting and the choice of polynomial vs exponential time schedules.
Problem
Existing theory of diffusion models mostly analyzes sampling assuming the score is already accurately learned, and training theory is limited; the interplay of training and sampling design choices (noise distribution, loss weighting, time and variance schedules) is not understood.
Approach
Combines an exponential convergence result for gradient descent on denoising score matching (deep ReLU network, lower bound on gradient via a high-dimensional geometric argument) with a KL sampling error bound for variance exploding SDEs under only a finite second moment assumption, yielding an end-to-end error bound used to compare schedules and weightings.
Key results
- Proves exponential convergence of GD on the discretized denoising score matching loss for a deep ReLU network with only the last-layer-fixed setting, with width requirement polynomial in n, N, d, L, T/t0.
- Extends sampling KL error analysis to VESDE under only finite second moment, obtaining almost linear dependence on data dimension under the exponential time schedule.
- Theory qualitatively supports EDM's bell-shaped total weighting; polynomial schedule preferred when score error dominates, exponential schedule preferred when discretization/initialization error dominates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the polynomial rather than exponential schedule (ablation) | Proposed VESDE sampling analysis · Score error dominates | score error factor: Exp.'s result ... is larger ... than the Poly.'s result | Helped |
| Use the exponential rather than polynomial schedule (ablation) | Proposed VESDE sampling analysis · Sampling error dominates | iteration complexity: poly-logarithmic dependence ... better than ... polynomially dependent | Helped |
| Use bell-shaped total weighting, as in EDM | Denoising score matching training · Denoising score matching training | gradient descent convergence rate | Helped |
| Use the exponential schedule instead of the polynomial schedule (ablation) | Proposed VESDE sampling analysis · Sampling error dominates | iteration complexity: polynomially dependent on \bar{\sigma}_{\max}/\bar{\sigma}_{\min} -> poly-logarithmic dependence on \bar{\sigma}_{\max}/ | Helped |
Surprising findings
- The analysis cannot make the training loss arbitrarily small: the population denoising loss is bounded below by a positive constant, so the optimization error and the bound are in tension with statistical error for fixed width.
Limitations
- Uses deep ReLU networks rather than U-Nets or transformers used in practice.
- Does not analyze generalization error; only optimization and sampling error are covered.
- Comparisons with practical designs are qualitative and limited to two schedules in extreme regimes.
Experimental setup
Metrics: KL divergence (theoretical bound) training loss convergence rate
Baselines: EDM polynomial time schedule exponential time schedule
Builds on: EDM Score Matching with Langevin dynamics stochastic localization sampling analysis semi-smoothness convergence framework for overparameterized networks
Closest papers by meaning
- Towards Non-Asymptotic Convergence for Diffusion-Based Generative Models (ICLR 2024 · similarity 0.91)
- One-Step Diffusion Distillation through Score Implicit Matching (NeurIPS 2024 · similarity 0.91)
- Training Diffusion-based Generative Models with Limited Data (ICML 2025 · similarity 0.91)
- Nearest Neighbour Score Estimators for Diffusion Generative Models (ICML 2024 · similarity 0.90)
- Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion (ICML 2025 · similarity 0.90)
- CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.