Research map / Image generation / Fast diffusion sampling
A solvable model of learning generative diffusion: theory and insights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a tight high-dimensional asymptotic characterization of a two-layer tied denoising autoencoder trained by online SGD as a stochastic-interpolant generative model on low-dimensional Gaussian-mixture manifolds, and uses it to explain mode collapse and model collapse under retraining.
Problem
Understanding precisely which density a finite-width, SGD-trained denoising autoencoder generates in a diffusion/flow model, including failure modes such as mode collapse and model collapse, beyond bounding distances to the target.
Approach
Uses a summary-statistics ODE analysis of online SGD for a tied two-layer DAE with trainable skip connection inside a stochastic interpolant framework, yielding a low-dimensional SDE that exactly describes projections of the generated density; the skip connection's limited strength causes variance shrinkage (mode collapse), which compounds across generations.
Key results
- Theoretical low-dimensional predictions match finite-dimension (d=1000) simulations for a 2-mode Gaussian mixture, with Hellinger distance to target decreasing monotonically over training time.
- Skip connection converges to b_tau = Lambda E[beta]/(Lambda E[beta^2]+E[alpha^2]) (up to b_0 decay), so for small average covariance eigenvalue Lambda the generated modes have small variance, producing mode collapse on MNIST sevens.
- Iterating the characterization across generations shows the generated density's variance shrinks further and loses structure by generation g=2 (model collapse).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the DAE generative model longer | DAE diffusion model on a 2-mode Gaussian mixture · 2-mode Gaussian mixture | Hellinger distance | Helped |
| Train the DAE generative model longer | DAE diffusion model on a 2-mode Gaussian mixture · 2-mode Gaussian mixture | recovery of bimodal structure | Helped |
| Train the DAE generative model longer | DAE diffusion model on Gaussian-approximated MNIST sevens · MNIST sevens covariance approximation | variance match | Mixed |
| Retrain on synthetic data from the previous generation | DAE diffusion model on Gaussian-approximated MNIST sevens · Successive synthetic-data generations | variance and meaningful structure | Hurt |
| Include a time encoding scheme in the DAE (ablation) | Two-layer DAE generative model · All probed settings | model behavior | No clear effect |
| Use tied rather than untied encoder and decoder weights (ablation) | Two-layer DAE generative model · Generative-model experiments | phenomenology | No clear effect |
Surprising findings
- The skip connection strength can become of the same order as the average covariance eigenvalue, so the learned generative variance shrinks for real data with small average eigenvalues, causing mode collapse.
- Training the generative model on its own synthetic outputs progressively degrades the distribution (model collapse), with the second generation devoid of meaningful structure.
Limitations
- Analysis restricted to jointly diagonalizable cluster covariances and to finite-dimensional manifold ambient span with tied, shallow two-layer DAE.
- Experiments limited to small networks (r=1 or r=2) and simple datasets, with real data approximated by a Gaussian for MNIST.
Experimental setup
Datasets: MNIST FashionMNIST
Benchmarks: MNIST FashionMNIST
Metrics: Hellinger distance agreement between theoretical density predictions and simula
Builds on: stochastic interpolants tight characterization of autoencoder learning (Bonnaire et al.) summary-statistics ODE analysis of SGD (Saad & Solla style)
Closest papers by meaning
- Towards Non-Asymptotic Convergence for Diffusion-Based Generative Models (ICLR 2024 · similarity 0.90)
- Analysis of Learning a Flow-based Generative Model from Limited Sample Complexity (ICLR 2024 · similarity 0.89)
- Shallow diffusion networks provably learn hidden low-dimensional structure (ICLR 2025 · similarity 0.89)
- Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modelling (NeurIPS 2024 · similarity 0.89)
- Evaluating the design space of diffusion-based generative models (NeurIPS 2024 · similarity 0.89)
- A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.