Research map / Learning theory and algorithms / Kernels and regression theory
Model Collapse Demystified: The Case of Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives analytic test-error formulas for ridge regression trained on recursively synthesized labels from linear/OLS generators, showing model collapse as additive bias and n-scaled variance terms, and proposes adaptive regularization to mitigate it.
Problem
Understanding theoretically why recursively training regression models on their own synthesized outputs degrades test error ("model collapse"), and how regularization affects it.
Approach
Models a self-consuming loop in Gaussian-design linear/kernel ridge regression where each generation is an OLS fit on labels from the previous generation; derives exact (RMT-limit) test error decomposition into clean error, a bias increase, and an n-proportional noise term, and shows adaptive ridge regularization λ∝T^{-ℓ} with a sample-size-dependent exponent can mitigate collapse under power-law
Key results
- Test error under n-fold synthetic retraining equals clean error plus a bias increase and n*sigma_0^2*rho, matching simulations for d=300.
- Under power-law spectra, model collapse modifies scaling laws; optimal exponent ell_star=(b-a)ell_crit yields rate T^{-(b-a)c} versus clean rate T^{-2beta r}.
- With dependent design matrices and T0<d, the bias increase stays constant in n, while with independent design matrices it grows as (1-phi_0^{-n})||w0||^2 and leads to collapse to zero.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on n generations of synthesized labels instead of clean labels | Ridge regression · Gaussian regression, T_0 ≥ d+2 | test error: E_{test}^{clean} + nσ_0^2ρ | Hurt |
| Use independent design matrices across generations rather than reusing X_0 (ablation) | Noiseless over-parameterized synthetic data generator · Isotropic regression, T_0 < d | test error bias: (1-φ_0^{-n})||w_0||^2 | Hurt |
| Reuse the same design matrix X_0 across generations rather than resampling it (ablation) | Noiseless over-parameterized synthetic data generator · Isotropic regression, T_0 < d | test error bias: ||w_0||^2(1-1/φ_0) | Hurt |
| Increase the number of generations of fake data | Ridge regression with under-parameterized fake-data generator · Gaussian regression | test error: nσ_0^2ρ | Hurt |
| Use adaptive regularization exponent ℓ⋆ instead of the clean-data exponent ℓ_crit (ablation) | Downstream ridge regression · Power-law covariance, noiseless regime | test error: T^{-(b-a)c} | Helped |
| Use the clean-data regularization exponent ℓ_crit on fake data (ablation) | Downstream ridge regression · Power-law covariance, noiseless regime | test error: T^{-(b-ℓ_crit/β-a)} | Hurt |
| Increase the fake-data generator's training sample size across generations (ablation) | Iterative fake-data generator · Gaussian regression | test error: log n · E_test(ŵ_0^{pred}) | Helped |
| Set the fake-data generator sample size to at least the input dimension (ablation) | Iterative fake-data generator · Gaussian regression | test error bias | Helped |
| Increase the downstream sample size T under iterative data relabeling | Kernel ridge regression · MNIST, RBF and polynomial kernels | test error | Hurt |
Surprising findings
- With independent design matrices and over-parametrized generators, the fake-data labeller collapses to zero, so the downstream error approaches the null predictor error even without label noise.
- Using the classical clean-data optimal regularization can cause catastrophic collapse where test error diverges with T.
Limitations
- Analysis restricted to linear/kernel ridge regression with Gaussian design and OLS generators, not deep networks or LLMs
- Many results are RMT-limit approximations (≃) rather than exact finite-sample formulas
- Mitigation requires knowing or tuning the sample-size-dependent regularization exponent
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: test error (excess risk)
Baselines: clean-data ridge regression (n=0)
Builds on: ridge regression bias-variance decomposition power-law scaling laws for kernel regression anisotropic local random matrix theory
Closest papers by meaning
- Strong Model Collapse (ICLR 2025 · similarity 0.91)
- When Models Don’t Collapse: On the Consistency of Iterative MLE (NeurIPS 2025 · similarity 0.90)
- A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective (NeurIPS 2025 · similarity 0.89)
- The Prevalence of Neural Collapse in Neural Multivariate Regression (NeurIPS 2024 · similarity 0.88)
- A Tale of Tails: Model Collapse as a Change of Scaling Laws (ICML 2024 · similarity 0.87)
- Learning with Synthetic Data via SGD in High-Dimensional Linear Regression (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.