Research map / Learning theory and algorithms / Sample complexity
Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses random matrix theory to analyze a ridge classifier trained on mixed real and verified (pruned) synthetic Gaussian-mixture data with both label and feature (distribution-shift) noise, deriving a closed-form test accuracy and a smooth phase transition in label noise.
Problem
Prior theory on training with verified synthetic data models only label noise and studies infinite-sample limits, ignoring feature distribution shift from finite-sample generative models in high dimensions.
Approach
Model synthetic data as Gaussian samples with statistics estimated from a finite real subset, so covariance estimation is inconsistent when p/n has a fixed ratio; use random matrix theory deterministic equivalents to derive the classifier's asymptotic accuracy under label noise and pruning, yielding a smooth rather than sharp phase transition.
Key results
- Theory predicts a smooth phase transition in classifier accuracy vs label noise in high dimension, recovering the sharp transition of Feng et al. as the sample-ratio parameter goes to zero.
- Under high dimension, even perfectly verified synthetic labels cause a performance drop due to feature distribution shift; synthetic data helps only when verification is accurate (strong supervision yields monotonic gains).
- Experiments on LLM safety alignment and Q&A generation show synthetic data from Llama3.1-8B-Instruct gives a better boost than from Gemma-2-2B-it as synthetic samples increase.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add synthetic data with strong verification | Ridge classifier on Amazon Reviews and one-hidden-layer ReLU network on MNIST · Amazon Reviews and MNIST | test accuracy | Helped |
| add synthetic data with weak supervision | Ridge classifier on Amazon Reviews and one-hidden-layer ReLU network on MNIST · Amazon Reviews and MNIST | test accuracy | Hurt |
| use a higher-quality synthetic-data generator | one-hidden-layer MLP on MNIST · MNIST | test accuracy | Helped |
| generate synthetic Q&A with Llama3.1-8B-Instruct instead of Gemma-2-2B-it | LLM Q&A Safety Generation · Q&A safety test set | test accuracy | Helped |
| increase the amount of synthetic data under weak supervision | models trained on real and synthetic data · Amazon Reviews and MNIST | test accuracy | Mixed |
| include feature distribution shift from finite-sample generation | classifier trained on mixed real and synthetic data · high-dimensional classification | test accuracy | Hurt |
| increase synthetic label noise in the synthetic-only setting | Ridge classifier trained only on synthetic data · synthetic-only classification | accuracy | Hurt |
Surprising findings
- Synthetic data can hurt performance in high dimension even when labels are noise-free, due to covariance estimation inconsistency causing feature distribution shift.
- Weak verification with label noise causes significant performance drops, with gains only visible at very large synthetic sample sizes.
Limitations
- Verification limited to labels; feature verification not studied.
- Analysis restricted to Gaussian data distributions and linear (ridge) classifiers.
- Results on LLMs are limited to a few models and safety tasks.
Experimental setup
Datasets: Amazon Reviews MNIST Anthropic HH-RLHF (safety subset) PKU-SafeRLHF ALERT 12k human Q&A safety dataset
Benchmarks: Amazon Reviews MNIST ALERT Q&A safety test set (2.8k)
Metrics: test accuracy safety score (percentage of safe answers)
Baselines: Feng et al. (2024) sharp-transition analysis
Builds on: Feng et al. (2024) synthetic data with verification Marchenko-Pastur law Gaussian mixture classification direct preference optimization
Closest papers by meaning
- Improving the Scaling Laws of Synthetic Data with Deliberate Practice (ICML 2025 · similarity 0.89)
- Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective (ICLR 2025 · similarity 0.88)
- Valid Inference with Imperfect Synthetic Data (NeurIPS 2025 · similarity 0.87)
- Learning with Synthetic Data via SGD in High-Dimensional Linear Regression (NeurIPS 2026 · similarity 0.87)
- Provably Improving Generalization of Few-shot models with Synthetic Data (ICML 2025 · similarity 0.87)
- How to Synthesize Text Data without Model Collapse? (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.