Research map / Probabilistic and scientific modeling / Molecules and proteins
Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that well-generalizing diffusion models have increasingly linear denoisers whose best linear approximation (via linear distillation) matches the Gaussian (empirical mean/covariance) optimal denoiser, indicating a Gaussian inductive bias.
Problem
Understanding why diffusion models generalize beyond training data rather than memorizing it, given that the theoretically optimal denoiser over an empirical dataset only reproduces training samples.
Approach
Measure denoiser linearity across noise levels, distill each nonlinear denoiser into a linear model (with bias), and show the distilled linear model matches the Gaussian optimal denoiser defined by the training set's empirical mean and covariance; the Gaussian bias emerges in the generalization regime (large data, small model, or early training).
Key results
- Distilled linear denoisers approximate nonlinear EDM denoisers with RMSE below 0.05 in the high-noise regime, and the Gaussian denoiser tracks the linear model closely across noise levels.
- Linearity score of EDM-VE denoisers stays above 0.96 even for the least-linear setting alpha=beta=1/sqrt(2), and linearity and generalization increase with dataset size.
- Smaller models trained on 1094 FFHQ images and early-stopped overparameterized models move toward Gaussian denoisers, while memorizing models diverge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase training dataset size at fixed model scale | EDM diffusion models with channel size 128 · FFHQ, intermediate noise variance [0.116,20] | score approximation error to Gaussian denoiser | Helped |
| Decrease model scale at fixed dataset size | EDM diffusion models trained on 1094 images · FFHQ, intermediate noise variance [0.116,20] | score discrepancy to Gaussian denoiser | Helped |
| Stop training at an early epoch rather than continuing toward convergence | Overparameterized diffusion model with channel size 128 · FFHQ, 1094 images | score difference to Gaussian denoiser | Mixed |
| Train on non-overlapping datasets of 35000 images each | Channel-size-128 diffusion models · Strong generalization on S1 and S2 | similarity of generated images across datasets | Helped |
| Use a smaller dataset of 1094 rather than 35000 images | Channel-size-128 diffusion models · Strong generalization on FFHQ | perceptual quality | Hurt |
| Use Gaussian denoisers instead of nonlinear diffusion denoisers | Diffusion image generation · Complex datasets such as CIFAR-10 | perceptual quality | Hurt |
| Use linear distilled or Gaussian denoisers instead of the multi-delta denoiser | Diffusion denoising · Intermediate-noise regime [0.1,20] | score approximation error | Helped |
| Use EDM-VE configuration with tailored skip and output scaling instead of Baseline-VE | VE diffusion models · Diffusion denoising and sampling | linearity | Helped |
Surprising findings
- Optimal linear denoisers are nearly identical to Gaussian denoisers, so a linear model reproduces much of the generated image structure of nonlinear diffusion models.
- Gaussian-based samples generated from a covariance estimated from only ~1094 images closely match those from models trained on 35000 images, contrary to prior claims that strong generalization requires >10^5 images.
Limitations
- A substantial gap remains between Gaussian/linear denoisers and nonlinear diffusion models, especially in intermediate noise levels, leading to lower perceptual quality on complex datasets like CIFAR-10.
- The mechanism by which gradient-trained nonlinear diffusion models become linear is not explained.
- Low-noise regime denoisers approximate identity mapping, so linearity analysis there is ambiguous.
Experimental setup
Datasets: FFHQ CIFAR-10 AFHQ LSUN-Churches
Benchmarks: FFHQ CIFAR-10 AFHQ LSUN-Churches
Metrics: linearity score (cosine similarity) NMSE score-difference RMSE generalization (GL) score FID
Baselines: Multi-delta (optimal empirical) denoiser Gaussian denoiser Baseline-VE
Builds on: EDM Wiener filter Gaussian score approximation (Wang & Vastola) memorization in diffusion models (Karras et al.)
Closest papers by meaning
- Generalization through variance: how noise shapes inductive biases in diffusion models (ICLR 2025 · similarity 0.95)
- Generalization Dynamics of Linear Diffusion Models (NeurIPS 2026 · similarity 0.92)
- Understanding diffusion models requires rethinking (again) generalization (NeurIPS 2026 · similarity 0.92)
- Towards a Mechanistic Explanation of Diffusion Model Generalization (ICML 2025 · similarity 0.92)
- On Inductive Biases That Enable Generalization in Diffusion Transformers (NeurIPS 2025 · similarity 0.91)
- An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.