Research map / Image generation / Fast diffusion sampling
On the Memorization of Consistency Distillation for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that consistency distillation of a memorizing diffusion teacher (EDM on CIFAR-10/ImageNet, SD v1.5) yields students with much lower memorization while preserving or improving FID/CLIP, and explains this with a random-feature model whose distillation curvature suppresses memorization-associated eigenmodes.
Problem
How does distillation (specifically consistency distillation) reshape memorization inherited from a teacher diffusion model, and does it trade off sample quality?
Approach
Empirically, consistency distillation of a memorizing teacher reduces memorization in the student under matched quality; theoretically, a random-feature neural network analysis shows the one-step CD curvature operator has a non-isotropic term that assigns near-zero positive update to memorization-associated eigenmodes of the teacher curvature.
Key results
- On CIFAR-10 (3000-data), the student reduces SSCD memorization from 21.68% (teacher) to 2.83% with FID 20.82 vs teacher 21.78.
- On ImageNet 10000-data, student FID 21.56 vs teacher 27.38 while SSCD memorization drops from 18.57% to 1.68%.
- On Stable Diffusion v1.5 (3x extra), SSCD-max drops from 0.6046 to 0.5774 with CLIP 0.2209 vs teacher 0.2274.
Surprising findings
- On ImageNet 7000-data, the student's FID (28.65) is worse than the teacher's (20.14) despite much lower memorization.
- Continuous-time consistency distillation produces blurred or failed generations in low-data regimes, whereas the discrete objective is stable.
Limitations
- Theory relies on a random-feature model in a high-dimensional asymptotic regime with a one-step, small-step-size approximation.
- FID does not improve for severely memorizing teachers and can degrade (e.g., ImageNet 7000-data).
- Memorization measured via l2 and SSCD thresholds, which are heuristic.
Experimental setup
Datasets: CIFAR-10 ImageNet COCO
Benchmarks: CIFAR-10 ImageNet Stable Diffusion v1.5 memorization protocol
Metrics: FID l2 memorization ratio SSCD memorization ratio SSCD p95 similarity CLIP score SSCD mean/max similarity
Baselines: progressive distillation DMD continuous-time consistency distillation EDM teacher
Builds on: consistency models EDM latent consistency models progressive distillation random feature neural networks
Closest papers by meaning
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.92)
- On the Edge of Memorization in Diffusion Models (NeurIPS 2025 · similarity 0.92)
- Detecting, Explaining, and Mitigating Memorization in Diffusion Models (ICLR 2024 · similarity 0.91)
- Understanding diffusion models requires rethinking (again) generalization (NeurIPS 2026 · similarity 0.90)
- How Diffusion Models Memorize (NeurIPS 2026 · similarity 0.90)
- Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.