Research map / Image generation / Autoregressive and latent generation
A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that recursive diffusion-model training shifts from generalization to memorization as dataset entropy falls, and proposes entropy-based (greedy farthest-point and threshold-decay) data selection to slow collapse.
Problem
Model collapse in diffusion models under self-consuming (recursive synthetic-data) training is usually characterized by variance shrinkage or population risk, which miss practical symptoms such as rapid loss of image diversity and memorization of training images.
Approach
Identifies a generalization-to-memorization transition across self-consuming iterations, links it to a strong correlation between training-set differential entropy (Kozachenko-Leonenko estimator) and the generalization score, and uses entropy-motivated greedy farthest-point or threshold-decay selection in DINOv2 feature space to curate training subsets.
Key results
- Entropy of the training set has a Pearson correlation of 0.91 with the log generalization score across dataset sizes and iterations.
- On CIFAR-10 under the accumulate-subsample paradigm, the vanilla baseline reaches FID 75.7 at iteration 8 while Greedy Selection reduces it to 44.7.
- On FFHQ under accumulate-subsample, Greedy Selection selects 65% real images at iteration 8 versus 12.5% for vanilla subsampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Greedy Selection to accumulate-subsample training | Vanilla accumulate-subsample · CIFAR-10, iteration 8 | FID: 75.7 -> 44.7 | Helped |
| Add entropy-based data selection | Vanilla replace and accumulate-subsample · CIFAR-10, FFHQ, and MNIST | generalization score | Helped |
| Add entropy-based data selection | Vanilla accumulate-subsample · CIFAR-10 and other evaluated datasets | FID | Helped |
| Use all 2N generated images for training without filtering instead of the vanilla N-image replace setting (ablation) | Vanilla replace · CIFAR-10, recursive training | FID | Helped |
| Filter 2N generated images to train on a selected N-image subset instead of using all 2N images (ablation) | Training on 2N generated images · CIFAR-10, recursive training | FID | Helped |
| Set filter ratio to 0.6 (ablation) | Data selection from 2N generated images · CIFAR-10 | FID: 40.0 | Helped |
| Set filter ratio to 0.8 rather than 0.6 (ablation) | Data selection from 2N generated images · CIFAR-10 | FID: 40.0 -> 38.2 | Helped |
| Set filter ratio to 1.0 rather than 0.8 (ablation) | Data selection from 2N generated images · CIFAR-10 | FID: 38.2 -> 49.5 | Hurt |
| Vary the Threshold Decay Filter decay rate (ablation) | Threshold Decay Filter with accumulate-subsample · CIFAR-10 | FID | Helped |
Surprising findings
- Variance of the training set correlates much more weakly with generalization than entropy does.
- Training on all 2N generated images without filtering is worse than selecting N of them, showing selection beats simply adding more synthetic data.
Limitations
- Greedy selection is computationally expensive.
- Does not rigorously explain why entropy collapses.
- Accumulate-paradigm experiments run for fewer than 10 iterations, so convergence behavior is unknown.
Experimental setup
Datasets: CIFAR-10 FFHQ MNIST
Benchmarks: CIFAR-10 FFHQ MNIST
Metrics: FID generalization score differential entropy (KL estimator) Pearson correlation
Baselines: vanilla replace paradigm vanilla accumulate-subsample paradigm classifier-free guidance without filtering
Builds on: DDPM model collapse (Shumailov et al.) generalization score (Zhang et al. 2024) classifier-free guidance farthest-point sampling
Closest papers by meaning
- On the Edge of Memorization in Diffusion Models (NeurIPS 2025 · similarity 0.92)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.91)
- Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.90)
- Understanding diffusion models requires rethinking (again) generalization (NeurIPS 2026 · similarity 0.90)
- A Tale of Tails: Model Collapse as a Change of Scaling Laws (ICML 2024 · similarity 0.89)
- On the Memorization of Consistency Distillation for Diffusion Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.