Research map / Image generation / Fast diffusion sampling
Generalization through variance: how noise shapes inductive biases in diffusion models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses a physics-inspired MSR path-integral analysis of PF-ODE sampling with noisy DSM training to show that diffusion models generalize beyond training points via a variance-induced (V-kernel) noise term that fills gaps between training examples.
Problem
Explaining when and how diffusion models generalize beyond their training set, given that the DSM optimum is the training distribution's score and networks are expressive enough to learn it.
Approach
The DSM target (proxy score) is noisy with covariance concentrated in boundary regions between training examples; averaging PF-ODE sampling over score-estimator randomness via a stochastic path integral yields an effective SDE whose noise (V-kernel) fills gaps, with feature and model-capacity factors modulating it.
Key results
- Proves that the naive score estimator (memorizing training data) still generalizes via an effective SDE with V-kernel proportional to the proxy score covariance, which is concentrated in boundary regions between training examples.
- Shows expressive linear models and lazy infinite-width NTK networks asymptotically generalize, with V-kernel noise scaling with F/P (model capacity ratio) and feature-related spectral biases.
- Small-noise semiclassical approximation gives learned distribution approx equal to epsilon-corrupted data distribution times a curvature factor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the noisy proxy-score target in DSM instead of the true score | Diffusion models trained with DSM · Finite training distributions | generalization beyond training examples | Helped |
| Remove proxy-score variance by setting the V-kernel to zero (ablation) | Diffusion models with an unbiased score estimator · Finite training distributions | generalization beyond training examples | Hurt |
| Use a naive score estimator that directly samples proxy scores | PF-ODE sampling · Discrete training distributions | generalization beyond training examples | Helped |
| Add feature-related inductive biases to the proxy-score covariance | Linear score models · Boundary regions and atypical feature values | effective sampling noise | Mixed |
| Vary training time in the lazy infinite-width network (ablation) | Lazy infinite-width neural networks · Diffusion-model generalization | feature-related inductive bias | Mixed |
| Increase both the time cutoff epsilon and the feature-to-sample ratio F/P (ablation) | Linear model with Gaussian features · One-dimensional data {-1, 0, 1} | difference between the average learned distribution and the training distributio | Helped |
| Use proxy-score variance to perturb sampling in boundary regions | Naive score estimator · Boundary regions between training examples | probability associated with boundary regions | Hurt |
| Use time-dependent features that create temporal autocorrelation in the V-kernel | Diffusion models with feature-based score estimators · Boundary regions between training examples | time spent in boundary regions | Helped |
| Use a single training example, making proxy-score covariance zero (ablation) | Diffusion models · Training distribution with M = 1 | generalization beyond the training example | Hurt |
| Use Gaussian rather than Fourier features, or change the data orientation (ablation) | Linear score models · Two-dimensional square-vertex data | shape of filled-in gaps | Mixed |
| Blend modes through generalization without conditioning | Unconditional diffusion models · MNIST digit images | generation of non-digit outputs | Hurt |
Surprising findings
- Diffusion models that exactly memorize training data via the naive noisy score estimator still generalize, because the proxy score variance injects noise.
- Generalization through variance can reduce probability in boundary regions in its naive form, since extra noise makes dynamics spend less time there.
Limitations
- Only unconditional, non-latent models; no text conditioning or attention layers.
- Realistic architectures (U-Nets) and rich learning dynamics not analyzed; only lazy/linear regimes.
- Neglects alternative training objectives (sliced score matching) and numerical integration error.
Experimental setup
Datasets: 1D toy data {-1,0,1} 2D point-cloud toy distributions (square)
Metrics: average learned distribution vs true distribution (qualitati
Baselines: naive score estimator true score (PF-ODE on training distribution)
Builds on: denoising score matching probability flow ODE Martin-Siggia-Rose path integral neural tangent kernel
Closest papers by meaning
- Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure (NeurIPS 2024 · similarity 0.95)
- Generalization Dynamics of Linear Diffusion Models (NeurIPS 2026 · similarity 0.92)
- Towards a Mechanistic Explanation of Diffusion Model Generalization (ICML 2025 · similarity 0.92)
- Understanding diffusion models requires rethinking (again) generalization (NeurIPS 2026 · similarity 0.91)
- Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold (NeurIPS 2026 · similarity 0.90)
- An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.