Research map / Architectures and efficiency / RNNs and normalization
Strong Model Collapse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that model collapse from synthetic data persists even when mixed with real data (strong model collapse), and that larger models can amplify collapse, using high-dimensional linear and random-projection regression plus GPT-2 language experiments.
Problem
Model collapse, the degradation of models trained on synthetic data, is studied in a scaling-laws setting where real and synthetic data are mixed; it is unclear whether mixing can avoid collapse and how model size affects it.
Approach
Derives a new bias-variance decomposition with an extra term from synthetic-real distribution mismatch, using operator-valued free probability to get exact high-dimensional test error for linear and random-projection models; shows the extra term does not vanish with a small synthetic fraction and that model size can amplify or mitigate collapse.
Key results
- Theory shows that for small feature dimension the test error plateaus at p2^2 c^2 whenever synthetic fraction p2 stays bounded away from zero, i.e. strong model collapse.
- In random-projection experiments, larger models are better when synthetic data is high quality but intermediate-size models are optimal for lower-quality synthetic data; beyond interpolation, larger models can mitigate collapse.
- On MNIST and GPT-2 (BabiStories) experiments, performance degrades as synthetic fraction grows and only recovers as p2 approaches 0; in fully trained MNIST networks larger models collapse more severely when trained solely on synthetic data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the synthetic-data proportion | Two-layer neural networks on MNIST · MNIST test set | test error | Hurt |
| Increase the synthetic-data proportion | GPT-2-small on BabiStories · BabiStories validation set | perplexity | Hurt |
| Increase model size | Fully trained two-layer neural networks on MNIST · MNIST test set; training sets of 10,000 to 50,000 samples | test error | Hurt |
| Increase model size from 12 to 18 or 24 layers | GPT-2 on BabiStories · BabiStories validation set; below 1 × 10^10 training tokens | test loss | Helped |
| Increase model size from 12 to 18 or 24 layers | GPT-2 on BabiStories · BabiStories validation set; above 3 × 10^10 training tokens | test loss | Hurt |
| Use a fixed nonzero weight for synthetic data in weighted single-step mixing | Classical linear model · Isotropic regression, small φ = d/n | test error | Hurt |
| Use iterative mixing instead of real data alone | Classical linear model · Isotropic regression | test error | Hurt |
| Iteratively mix synthetic and real data for t iterations while keeping real-data proportion non-vanishing | Classical linear model · Isotropic regression | scaling rate | No clear effect |
| Iteratively mix with a diminishing proportion of real data (p2 → 1) | Classical linear model · Isotropic regression | test error | Hurt |
Surprising findings
- Even a 1-in-1000 fraction of synthetic data can produce a performance plateau in the scaling-law regime.
- Iterative (multi-step) mixing mostly relies on real data and underperforms using real data alone, and with little real data it yields catastrophic collapse.
Limitations
- Theory is limited to linear and random-projection (linearized) models with Gaussian data and proportionate asymptotics.
- Fully trained neural network and language model results are empirical and not covered by the theory.
Experimental setup
Datasets: BabiStories MNIST synthetic Gaussian data
Benchmarks: MNIST BabiStories validation set
Metrics: test error (MSE) validation perplexity / test loss
Baselines: weighted data mixing iterative mixing / self-consuming retraining
Builds on: scaling laws for random feature models classical model collapse in linear regression operator-valued free probability
Closest papers by meaning
- A Tale of Tails: Model Collapse as a Change of Scaling Laws (ICML 2024 · similarity 0.93)
- Model Collapse Demystified: The Case of Regression (NeurIPS 2024 · similarity 0.91)
- Escaping Collapse: The Strength of Weak Data for Large Language Model Training (NeurIPS 2025 · similarity 0.90)
- Linguistic Collapse: Neural Collapse in (Large) Language Models (NeurIPS 2024 · similarity 0.90)
- Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification (ICLR 2025 · similarity 0.90)
- When Models Don’t Collapse: On the Consistency of Iterative MLE (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.