Research map / Image generation / Fast diffusion sampling
When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Nets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes probability flow and score flow of minimum-norm shallow ReLU denoisers, showing that on orthogonal or obtuse-simplex data the flows converge to training points, sums of training points ("virtual points"), or hyperbox boundary points, with diffusion time-scheduler early stopping driving generalization.
Problem
When does diffusion probability flow converge to training samples (memorization) versus other points on the data manifold, given neural denoisers that interpolate training data?
Approach
Characterizes the probability flow ODE of min-representation-cost shallow ReLU denoisers (from prior work) and compares it to a simpler score flow with fixed noise level; shows the diffusion time schedule causes early stopping that lets probability flow reach hyperbox boundary points and virtual sums of training points rather than only training points.
Key results
- Score flow from random initializations converges almost entirely to virtual points or hyperbox boundary; training-point convergence is rare.
- Probability flow with 500 samples in R^30 yields a large fraction of samples near virtual points, with fraction of training-point convergence decreasing as N grows.
- Generalization (virtual-point and boundary convergence) increases with the number of training points N.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of training samples | Probability flow with minimum-norm shallow denoisers · Orthogonal data | generalization, measured by convergence to virtual points or hyperbox boundaries | Helped |
| Use probability flow rather than score flow | Minimum-norm shallow denoisers · Orthogonal data | convergence to virtual points | Mixed |
| Add weight decay during denoiser training (ablation) | Shallow neural denoisers · Orthogonal data, probability flow | convergence to virtual points | Helped |
| Remove weight decay during denoiser training (ablation) | Shallow neural denoisers · Orthogonal data, probability flow | convergence to virtual points | Hurt |
| Add dropout during denoiser training (ablation) | Shallow neural denoisers · Orthogonal data, probability flow | MSE loss | Hurt |
| Add dropout during denoiser training (ablation) | Shallow neural denoisers · Orthogonal data, probability flow | samples generated outside the hyperbox boundary | Hurt |
| Oversample duplicate training points (ablation) | Diffusion sampling with shallow denoisers · Orthogonal data | memorization of duplicated training points | Hurt |
| Augment the training set with samples from the hyperbox boundary (ablation) | Minimum-norm shallow denoisers · Orthogonal data, N=40 and N=50 | convergence to the hyperbox boundary | Helped |
| Use more training samples in virtual-point combinations | Minimum-norm shallow denoiser · Orthogonal data, virtual points | percentage of virtual points stable in practice | Hurt |
Surprising findings
- Score flow converges almost never to training points itself, mostly to virtual sums or boundary points.
Limitations
- Assumes denoiser interpolates a full d-dimensional ball around each training point, while in practice the number of noisy samples is below d.
- Analysis restricted to a single hidden layer model.
- Results derived for VE process; variance-preserving case not analyzed.
Experimental setup
Datasets: synthetic orthonormal Gaussian-noise training data in R^30
Metrics: percentage of sampled points converging to virtual points, t
Baselines: Score flow
Builds on: Min-norm shallow ReLU denoisers (Ongie/Manor-style representation cost) Probability flow ODE Score-based generative modeling with SDEs
Closest papers by meaning
- An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.90)
- Generalization in diffusion models arises from geometry-adaptive harmonic representations (ICLR 2024 · similarity 0.90)
- Towards a Mechanistic Explanation of Diffusion Model Generalization (ICML 2025 · similarity 0.89)
- How Diffusion Models Memorize (NeurIPS 2026 · similarity 0.89)
- Understanding diffusion models requires rethinking (again) generalization (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.