Research map / Image generation / Text-to-image diffusion

Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences

NeurIPS 2024 · Image generation › Text-to-image diffusion · approach: Fairness and bias in generative models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically that curating synthetic samples with a K-choice Bradley-Terry model in iterative retraining acts as implicit RLHF-like preference optimization, maximizing expected reward and converging to the highest reward level set, with stability results when mixing real data, and illustrates bias amplification on synthetic data and CIFAR-10.

Problem

Self-consuming generative models retrained on web data that is partly synthetic and human-curated: how does curation of synthetic samples affect the distribution learned across retraining iterations, and does it stay stable?

Approach

Models curation of generated samples as a discrete K-choice Bradley-Terry selection; retraining on curated samples is shown to be an implicit preference optimization (equivalent to KL-regularized RLHF with beta = 1/t as K grows), with proven reward increase, variance collapse, and KL convergence to the top reward level set, plus stability bounds when real data is mixed in.

Key results

• Under the K-to-infinity limit, curated retraining converges to p_t(x) proportional to p_t(x) e^{r(x)} / E[e^{r}], and the expected reward increases monotonically with variance vanishing (Lemma on E[e^r] convergence to e^{r*}).
• Theorem: KL(p_* || p_t) goes to 0, so the loop converges to the highest reward level set reached at initialization.
• On CIFAR-10 with confidence-based reward, class proportions become increasingly heterogeneous (bias amplification), while mixing real data at lambda=1/2 keeps class proportions approximately uniform with reward still increasing.

Measured effects reported in the paper

Surprising findings

• Iterative curated retraining with K to infinity is equivalent to KL-regularized RLHF with regularization beta = 1/t, despite maximum likelihood on curated data being a different objective from direct reward maximization.
• Using classifier confidence as reward (uncorrelated with class label) still amplifies class imbalance across retraining steps on CIFAR-10.

Limitations

• Theory assumes infinite-capacity or idealized maximum likelihood retraining; the stability result for parametric models needs local assumptions near optimal parameters.
• No algorithm is proposed to mitigate bias amplification.
• Experiments are small-scale (synthetic 2D data and CIFAR-10 with a single run per figure), using a known reward model rather than real human preferences.

Experimental setup

Datasets: CIFAR-10 mixture of 8 Gaussians two moons

Benchmarks: CIFAR-10 class proportions and average reward across retraining steps synthetic Gaussian mixture and two moons retraining curves

Metrics: expected reward class proportions KL divergence

Baselines: Without-curation retraining (fully synthetic) Retraining on mixed real and synthetic data without curation

Builds on: RLHF Bradley-Terry model model collapse self-consuming loops (Alemohammad et al.) conditional flow matching DDPM

Closest papers by meaning

• On the Stability of Iterative Retraining of Generative Models on their own Data (ICLR 2024 · similarity 0.93)
• Self-Consuming Generative Models with Adversarially Curated Data (ICML 2025 · similarity 0.92)
• Self-Consuming Generative Models with Co-Evolving Human Preferences (NeurIPS 2026 · similarity 0.91)
• Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.90)
• Towards Theoretical Understandings of Self-Consuming Generative Models (ICML 2024 · similarity 0.89)
• A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops (ICLR 2025 · similarity 0.89)