Research map / Probabilistic and scientific modeling / Variational and generative sampling
Towards Theoretical Understandings of Self-Consuming Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives theoretical TV-distance bounds for diffusion models (one-hidden-layer random-feature score) and KDE trained in self-consuming loops under general, full-synthetic, and balanced data mixing, and shows a phase transition in error as synthetic data grows.
Problem
Understanding how training generative models recursively on mixtures of real and synthetic data from previous generations (self-consuming loops) affects the distribution learned by future models, which lacks rigorous theory beyond simple Gaussian toy models and parameter-level analyses.
Approach
Derive high-probability TV-distance bounds between future generations' learned distributions and the real data distribution, for diffusion models with a one-hidden-layer random-feature score network trained by gradient flow and for kernel density estimators, without assuming bounds on statistical or optimization errors; the recursion over mixing weights yields how real-data proportion, sample size
Key results
- For full synthetic cycles, TV error is controlled only if sample sizes grow quartically across generations (n_k = Omega((i sqrt(d)/eps)^4)) or the final generation includes real-data proportion alpha = Omega((i-1)/i).
- With a fixed real-data count n and synthetic count m = lambda n, the bound f(lambda,i) first increases and then decreases in lambda, a phase transition whose critical lambda grows with generation count i (numerically observed, no closed form).
- For general mixing with constant real-data fraction alpha, the error bound scales as (1-(1-alpha)^{i+1})/alpha times the per-generation statistical term, so constant real-data fraction keeps error controlled.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add real data at each generation with a constant proportion | Diffusion model self-consuming loop · General data cycle | TV distance: TV(p_{\theta_{i+1}},p_0) \lesssim \left(1-(1-\alpha)^{i+1}\right)\alpha^{-1} (n^{-\frac{1}{4}}\sqrt{d\log \frac{di}{\del | Helped |
| Use only latest-generation synthetic data, without real data | Diffusion model self-consuming loop · Full synthetic data cycle | TV distance: TV(p_{\theta_{i+1}},p_0)=\mathcal{O}((1-\alpha)i\epsilon) | Hurt |
| Grow training sample sizes quartically across generations | Diffusion model self-consuming loop · Full synthetic data cycle | TV distance: n_k = \Omega\left((i\sqrt{d}/\epsilon)^{4}\right); TV distance to \mathcal{O}(\epsilon) | Helped |
| Add a sufficiently large proportion of original data in the final generation | Diffusion model self-consuming loop · Full synthetic data cycle | TV distance: \alpha=\Omega(\frac{i-1}{i}) | Helped |
| Use a uniform mixture of real data and synthetic data from all previous generations | Diffusion model self-consuming loop · Balanced data cycle | TV distance: TV(p_{\theta_{i+1}},p_0)=\mathcal{O}(\epsilon) | Helped |
| Increase synthetic samples while keeping real sample count fixed | Diffusion model self-consuming loop · Fixed real data with synthetic augmentation | TV distance: TV(p_{\theta_{i+1}},p_0)=\mathcal{O}\left(\frac{1}{n^{1/4}}\frac{(1+\lambda)^{i+1}-\lambda^{i+1}}{(1+\lambda)^{i+\frac{1 | Mixed |
| Increase the synthetic-to-real sample ratio beyond the phase-transition threshold | Diffusion model self-consuming loop · Fixed real data with synthetic augmentation | TV distance: f converges to $(i+1)(1+\lambda)^{-1/4}$ | Helped |
| Use KDE with bandwidth h_i=n_i^{-1/(2s+2d)} | Kernel density estimation · One generation of the self-consuming loop | TV distance: TV(\widehat{p}_{i+1}, p_i)\leq \frac{1}{2}n_i^{-\frac{2s+d}{4s+4d}} (1+\gamma_{n_i})\sqrt{ \int K^2}\int \sqrt{p_i} +\fr | Helped |
Surprising findings
- Adding modest amounts of synthetic data can initially worsen the TV bound before it improves beyond a threshold, contrary to the monotone improvement seen in some experiments.
Limitations
- Analysis restricted to one-hidden-layer random-feature score networks and gradient-flow training, with bounds that hide constants and assume unbiased sampling.
- Full-synthetic results require quartic sample growth, which is impractical; the phase-transition threshold has no closed form and is found numerically.
- Theoretical bounds only, with no experiments on real generative models in this text.
Experimental setup
Metrics: total variation distance (theoretical bound)
Baselines: Bertrand et al. 2023 (self-consuming Gaussian analysis)
Builds on: Gradient-descent training analysis for diffusion models (Lemma 4/Theorem 1 cited Bertrand-style Gaussian self-consuming analysis (Bertrand et al. 2023) Alemohammad et al. 2023 self-consuming loop experiments McDiarmid's inequality
Closest papers by meaning
- On the Stability of Iterative Retraining of Generative Models on their own Data (ICLR 2024 · similarity 0.92)
- A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops (ICLR 2025 · similarity 0.90)
- Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.90)
- Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.89)
- Stability and Diversity of Networked Self-Consuming Generative Ecosystems (NeurIPS 2026 · similarity 0.88)
- Self-Correcting Self-Consuming Loops for Generative Model Training (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.