Research map / Learning theory and algorithms / Kernels and regression theory

Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

NeurIPS 2026 · Learning theory and algorithms › Kernels and regression theory · approach: High-dimensional asymptotic generalization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes one-pass SGD on high-dimensional linear regression with synthetic and real data under mixed and two-stage training, showing mixed training suffers strong model collapse while two-stage training avoids it.

Problem

How synthetic data affects finite-sample generalization of one-pass SGD in high-dimensional linear regression with model shift between real and synthetic labeling functions, and whether strong model collapse is inevitable under fixed synthetic proportion.

Approach

Decomposes SGD error under mixed training into fluctuation and persistent drift terms caused by source mismatch, shows a non-vanishing p^2 ||delta||^2 floor, and shows a two-stage curriculum (synthetic first, then real) removes the floor; adds random sketch scaling laws and an exact necessary-and-sufficient condition for two-stage benefit.

Key results

• Mixed training excess risk has a non-vanishing floor of order p^2 ||delta||_H^2 as M+N grows, while two-stage training's excess risk vanishes under the stated spectral condition (D_eff = o(N_eff)).
• Random sketch scaling law for mixed training is tight in the optimization-saturated regime: risk approx D^{1-a} + N_eff^{1/a-1} + p^2(1-D^{1-b}), so larger sketch dimension amplifies drift floor.
• Exact finite-sample condition: two-stage beats real-only iff ||E[w_M]-w_1*||_K^2 + tr(K Cov(w_M)) < ||w_0-w_1*||_K^2; in 1D, requires delta^2 < (w0-theta)^2 and M > sigma_2^2/((w0-theta)^2-delta^2).

Surprising findings

• Mixed training with iterate averaging exhibits the same p^2 error floor, indicating the collapse is not an artifact of last-iterate SGD.
• Two-stage training's upper bound cannot be matched termwise by a lower bound because cross terms may cancel.

Limitations

• Analysis restricted to one-pass SGD with shared covariates; joint covariate and model shift and multi-epoch SGD not covered.
• Mixed-training scaling law is tight only in the optimization-saturated regime; two-stage result is only an upper bound.
• Real-world experiments (CIFAR10 CNN, WikiText-103) are only briefly described and rely on simulated synthetic teacher data.

Experimental setup

Datasets: CIFAR10 WikiText-103

Benchmarks: CIFAR10 WikiText-103

Metrics: excess risk

Baselines: real-only training mixed training

Builds on: strong model collapse analysis (Dohmatob et al.) high-dimensional SGD analysis with geometric step decay random sketch scaling laws

Closest papers by meaning

• Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.89)
• Towards Theoretical Understandings of Self-Consuming Generative Models (ICML 2024 · similarity 0.88)
• Improved Scaling Laws in Linear Regression via Data Reuse (NeurIPS 2025 · similarity 0.88)
• When Models Don’t Collapse: On the Consistency of Iterative MLE (NeurIPS 2025 · similarity 0.87)
• Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory (ICLR 2025 · similarity 0.87)
• A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops (ICLR 2025 · similarity 0.87)