Research map / Image generation / Fast diffusion sampling
Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows via U-Net experiments on CelebA and a random-features analysis that diffusion models' memorization time scales linearly with training set size while generalization time stays constant, creating a widening early-stopping window.
Problem
Why diffusion models avoid memorizing training data despite being highly overparameterized, and how the training dynamics control the transition from generalization to memorization.
Approach
Identifies two training timescales: tau_gen (independent of n, when high-quality generalizing samples emerge) and tau_mem (proportional to n, when memorization sets in), so implicit dynamical regularization from training dynamics yields a growing generalization window with n; explained analytically via the spectrum of a random features model under gradient flow.
Key results
- Memorization time tau_mem scales linearly with training set size n (collapse at tau/n ~ 300) while generalization time tau_gen (~100K steps) is independent of n on CelebA U-Nets.
- In the random features model, timescales are set by inverse eigenvalues of the feature covariance; the bulk rho_2 gives tau_gen of order 1/Delta_t and the bulk rho_1 gives tau_mem of order psi_n/Delta_t.
- Scaling experiments show tau_gen proportional to W^-1 and tau_mem proportional to n W^-1, so the critical n at which the generalization window opens is roughly independent of model size p over the tested range.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the training set size n | CelebA U-Net score model · CelebA | memorization time: τ_mem ∝ n | Helped |
| increase the training set size n | CelebA U-Net score model · CelebA | generalization time: τ_gen (≈100K) | No clear effect |
| increase U-Net base width W (ablation) | CelebA U-Net score model · CelebA | generalization and memorization times: τ_gen ∝ W^{-1}; τ_mem ∝ nW^{-1} | Mixed |
| use full-batch updates (B=n) instead of fixed batch size (ablation) | CelebA U-Net score model · CelebA, n∈[128, 512] | memorization time scaling with n: τ_mem ∝ n | No clear effect |
| replace SGD with momentum by Adam (ablation) | CelebA U-Net, W=64 · CelebA | memorization time scaling with n: τ_mem ∝ n | No clear effect |
| replace SGD with momentum by Adam (ablation) | CelebA U-Net, W=64 · CelebA | optimization steps to reach the two timescales | Helped |
| increase the training set size n | Gaussian-mixture diffusion model · Gaussian mixture model | memorization time: τ_mem ∝ n | Helped |
| increase network width W (ablation) | Gaussian-mixture diffusion model · Gaussian mixture model | memorization time: τ_mem ∝ W^{-1} | Hurt |
| condition generation on the Gaussian-mixture class label (ablation) | Classifier-free guidance model · Conditional Gaussian mixture model | memorization time scaling with n: curves ... collapse perfectly when rescaling the training time by n | No clear effect |
Surprising findings
- Memorization delay with n is not explained by sample repetition: full-batch training still gives tau_mem proportional to n.
Limitations
- Theoretical analysis restricted to random features model with fixed diffusion time and SGD/gradient flow; not full diffusion training.
- Experiments explore only p between 1M and 16M parameters on CelebA 32x32 grayscale images.
- Results shown for SGD; Adam only checked in supplementary experiments and conditional settings only in toy GMM.
Experimental setup
Datasets: CelebA Gaussian mixture (synthetic)
Benchmarks: CelebA
Metrics: FID memorization fraction training loss test loss generalization loss KL divergence
Baselines: empirical score architectural regularization threshold n*(p)
Builds on: DDPM Random Features Neural Network Gaussian Equivalence Principle Adam
Closest papers by meaning
- Understanding diffusion models requires rethinking (again) generalization (NeurIPS 2026 · similarity 0.94)
- On the Edge of Memorization in Diffusion Models (NeurIPS 2025 · similarity 0.94)
- On the Memorization of Consistency Distillation for Diffusion Models (NeurIPS 2026 · similarity 0.92)
- How Diffusion Models Memorize (NeurIPS 2026 · similarity 0.92)
- Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion (ICML 2025 · similarity 0.92)
- A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.