Research map / Image generation / Fast diffusion sampling
On the Edge of Memorization in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a Gaussian-mixture "memorization laboratory" for diffusion models and predicts the memorization-generalization phase transition via a training-loss crossover between a partially memorizing and a generalizing denoiser, validated in trained models.
Problem
When do trained diffusion models memorize their training data versus generalize beyond it, and can this transition be predicted theoretically rather than through heuristic memorization metrics?
Approach
Model the data as Gaussian mixtures and the denoiser as a Gaussian-mixture MMSE denoiser with M components, so the class interpolates between the generalizing denoiser (M=K) and the memorizing denoiser (M=N). Hypothesize that a trained model memorizes iff the training loss of a partially memorizing denoiser (memorizing M samples) is below that of the generalizing denoiser; derive high-dimensional
Key results
- Trained denoisers show a phase transition from generalization to memorization as model size M grows, with memorization ratio rising to near 100%.
- The fitted loss weighting yields a predicted crossover M_pt approximately (4/5)N, linear in N, with train and test errors below 2e-4.
- The low-rank Gaussian mixture image model exhibits a qualitatively similar phase transition.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Predict the transition location using the derived approximate training losses | Trained denoisers on isotropic Gaussian mixtures · Isotropic Gaussian mixture | phase-transition location prediction error: Train and test errors are \(\leq 2 \times 10^{-4}\); within one or two \(M\)'s on average | Helped |
| Increase the number of components in the trained denoiser | Trained denoisers · Isotropic Gaussian mixture | memorization ratio | Mixed |
| Use the recovered loss-based transition estimate | Trained denoisers · Isotropic Gaussian mixture | memorization phase-transition location: \(M_{\mathrm{pt}}(N, d, K) \approx \tilde{M}_{\mathrm{pt}}(N, d, K, \tilde{\lambda}) = (4/5)N\) | Helped |
| Use a low-rank Gaussian mixture data model instead of the isotropic Gaussian mixture setting | Trained denoisers · Low-rank Gaussian image model | memorization phase-transition behavior | No clear effect |
Limitations
- Analysis restricted to isotropic or low-rank Gaussian mixture data, not large-scale natural image diffusion models.
- Theoretical asymptotics hold in high dimensions and only approximately match experiments at moderate dimension.
- Loss weighting lambda is fitted empirically rather than derived from first principles.
Experimental setup
Datasets: synthetic isotropic Gaussian mixture data FashionMNIST templates (low-rank Gaussian mixture)
Benchmarks: synthetic Gaussian mixture laboratory low-rank Gaussian mixture image model
Metrics: memorization ratio train loss test loss predicted vs. measured crossover M_pt/N train and test error of loss-weight fit
Baselines: generalizing denoiser memorizing denoiser partially memorizing denoiser
Builds on: DDIM Tweedie's identity Gaussian mixture denoisers memorization criterion of Carlini et al. / Somepalli et al.
Closest papers by meaning
- Understanding diffusion models requires rethinking (again) generalization (NeurIPS 2026 · similarity 0.95)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.94)
- Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion (ICML 2025 · similarity 0.93)
- A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective (NeurIPS 2025 · similarity 0.92)
- On the Memorization of Consistency Distillation for Diffusion Models (NeurIPS 2026 · similarity 0.92)
- How Diffusion Models Memorize (NeurIPS 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.