Research map / Probabilistic and scientific modeling / Variational and generative sampling
When Models Don’t Collapse: On the Consistency of Iterative MLE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of iterative MLE trained on accumulating real plus synthetic data, showing non-asymptotic consistency under standard smoothness/regularity assumptions and constructing counterexamples where collapse occurs quickly despite MLE consistency.
Problem
Whether model collapse is avoidable when MLE is iteratively retrained on accumulating data (real plus synthetic), and under which assumptions; prior analyses reached conflicting conclusions about its severity.
Approach
Uses a Taylor-expansion/telescoping argument on the stationary-point conditions of the cumulative log-likelihood to show that iterative MLE parameters stay within O(sqrt(log(d/δ)/n)) of ground truth when n is polylogarithmic in T; complements this with constructions of TV-consistent families where collapse happens after one or arbitrarily many iterations.
Key results
- Iterative MLE stays within C*sqrt(log(4d/δ)/n) of ground truth parameters for all T when n >= c(log T+1)^2 log^2(7dT/δ), so collapse can be avoided even as the real-data fraction vanishes.
- A TV-consistent family exists where TV(p_theta*, p_theta2) >= C with constant probability while TV(p_theta*, p_theta1) <= log(n)/n, so collapse can occur after a single iteration.
- For any increasing phi with phi(n)->infinity, a TV-consistent family exists where TV >= 3/8 occurs within O((1/δ)log(4/δ) max(phi(n),1)) iterations, e.g. O(log log n) iterations for phi = log log n.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Satisfy the stated regularity, smoothness, and Fisher information assumptions during accumulating-data iterative MLE | Iterative MLE · Iterative MLE, iteration T | parameter error: \norm{\thetait{T}-\theta^{\star}}\leq C \sqrt{\frac{\log\left(\frac{4d}{\delta}\right)}{n}} | Helped |
| Accumulate synthetic data with a TV-consistent but highly non-smooth distribution family | Iterative MLE · Accumulating-data iterative MLE, iteration 2 | total variation distance: \operatorname{TV}\left(p_{\theta^{\star}}, p_{\thetait{2}}\right) \geq C | Hurt |
| Use a fixed TV-consistent family that violates the positive-result assumptions | Iterative MLE · Accumulating-data iterative MLE, some iteration T | total variation distance: \operatorname{TV}\left(p_{\thetait{T}}, p_{\theta^{\star}}\right) \geq \frac{3}{8} | Hurt |
Surprising findings
- MLE consistency alone is insufficient to prevent rapid model collapse even when real data remains in the training set; collapse can occur after a single iteration for some families.
Limitations
- Assumptions (smoothness, bounded third derivatives, sub-Gaussian gradients, Fisher information lower bound) may be restrictive for neural networks.
- Analysis assumes exact MLE/stationary points; optimization error only empirically examined.
- Negative constructions are artificial non-smooth families and the gap between positive and negative assumptions remains.
Experimental setup
Metrics: parameter error norm total variation distance KL divergence
Builds on: Shumailov et al. model collapse asymptotic normality of MLE Bernstein-type matrix Azuma concentration
Closest papers by meaning
- Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.90)
- Model Collapse Demystified: The Case of Regression (NeurIPS 2024 · similarity 0.90)
- Strong Model Collapse (ICLR 2025 · similarity 0.89)
- A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops (ICLR 2025 · similarity 0.89)
- A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective (NeurIPS 2025 · similarity 0.89)
- A Tale of Tails: Model Collapse as a Change of Scaling Laws (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.