Research map / Image generation / Fast diffusion sampling

On the Stability of Iterative Retraining of Generative Models on their own Data

ICLR 2024 · Image generation › Fast diffusion sampling · approach: Distributional control of diffusion · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Develops a maximum-likelihood theory of iterative retraining of generative models on mixtures of real and self-generated data, proving stability when real data proportion is large enough and the initial model is close to the data distribution, and validates it with OTCFM, DDPM and EDM on CIFAR-10 and FFHQ-64.

Problem

Self-consuming generative models retrained on web data containing their own synthetic outputs may collapse; the paper asks whether training on mixed finite real and self-generated data changes performance and stability.

Approach

Frames iterative retraining as a maximum-likelihood fixed-point map over parameters with a real-data term and a synthetic-data regularizer weighted by lambda; shows local stability (Jacobian norm < 1) when lambda(1+L*eps/alpha) < 1/2 and the model is close to the data distribution, and extends to finite-sample error with a neighborhood bound.

Key results

• Under sufficient real-data proportion (small lambda), iterative retraining of EDM on CIFAR-10 and FFHQ-64 remains stable without FID divergence, whereas lambda=1 (equal real and synthetic) may cause FID to diverge.
• The theorem gives a Jacobian operator-norm bound lambda(alpha+eps L)/(alpha-lambda(alpha+eps L)) < 1 under lambda(1+L eps/alpha) < 1/2, implying linear local convergence to the fixed point.
• For a single multivariate Gaussian retrained only on its own samples, covariance collapses to zero at a linear rate alpha^t with alpha < 1.

Measured effects reported in the paper

Surprising findings

• Stability can be achieved without injecting fresh data, only by keeping a sufficient fraction of the original real data at each retraining step.
• For EDM, precision decreases while recall increases across retraining iterations, and the trend is reversed for OTCFM on CIFAR-10.

Limitations

• The sufficient stability condition may not be necessary; a gap remains between necessity of finite lambda and sufficiency lambda(1+L eps/alpha) < 1/2.
• Theory is local around the optimal parameters and relies on assumptions (Hessian Lipschitzness, strict local maximum) that are hard to verify for deep models.
• Experiments use a fixed budget of finetuning images per iteration and only image datasets at 64x64 resolution.

Experimental setup

Datasets: CIFAR-10 FFHQ-64 8 Gaussians (synthetic) two moons (synthetic)

Benchmarks: CIFAR-10 FFHQ-64

Metrics: FID precision recall

Baselines: OTCFM DDPM EDM

Builds on: denoising diffusion probabilistic models elucidating diffusion models conditional flow matching

Closest papers by meaning

• Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.93)
• Towards Theoretical Understandings of Self-Consuming Generative Models (ICML 2024 · similarity 0.92)
• Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.91)
• A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops (ICLR 2025 · similarity 0.90)
• The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better (NeurIPS 2024 · similarity 0.90)
• Stability and Diversity of Networked Self-Consuming Generative Ecosystems (NeurIPS 2026 · similarity 0.89)