Research map / Architectures and efficiency / RNNs and normalization
Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Deliberate Practice (DP), a loop where a diffusion model generates synthetic training data on demand, guided by the classifier's prediction entropy whenever validation accuracy plateaus, so that only hard examples are generated for training a classifier without real data.
Problem
Training classifiers purely on synthetic images from text-to-image models shows diminishing returns with dataset size, and generate-then-prune wastes compute on discarded samples.
Approach
Dynamically generate synthetic data directly from the entropy-pruned distribution using entropy-guided diffusion sampling, triggered by a patience mechanism on real validation accuracy, instead of generating a large static pool and pruning it; a random-matrix-theory analysis supports improved scaling from selecting hard examples.
Key results
- DP matches the best static ImageNet-100 accuracy (3M samples) with 400k samples, a 7.5x reduction in data.
- On ImageNet-1k, DP with 640k samples outperforms the best static setup of 13M samples (20x less data).
- DP reaches 74.3% real ImageNet-100 val accuracy with 1.9M synthetic samples versus 73.3% for static with 6.5M samples; on ImageNet-1k, 55.1% with 9.1M samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace static synthetic data generation with dynamic entropy-guided DP generation | ViT-B synthetic-data training · ImageNet-100 | Real Val. Acc.: 73.3 -> 74.3 | Helped |
| Replace static synthetic data generation with dynamic entropy-guided DP generation using fewer samples | ViT-B synthetic-data training · ImageNet-1k | Real Val. Acc.: 54 -> 55.1 | Helped |
| Add entropy guidance (ω=0.05) instead of regular sampling (ω=0) (ablation) | DP on ImageNet-100 · ImageNet-100 | Real Val. Acc.: 61.58 -> 68.04 | Helped |
| Set entropy-guidance coefficient to ω=0.03 instead of ω=0.05 (ablation) | DP on ImageNet-100 · ImageNet-100 | Real Val. Acc.: 68.04 -> 66.70 | Hurt |
| Set entropy-guidance coefficient to ω=0.07 instead of ω=0.05 (ablation) | DP on ImageNet-100 · ImageNet-100 | Real Val. Acc.: 68.04 -> 66.88 | Hurt |
| Use fixed rather than incremental maximum patience (ablation) | DP on ImageNet-100 · ImageNet-100 | Real Val. Acc.: 68.04 -> 67.22 | Hurt |
| Use non-uniform sampling rather than uniform sampling (ablation) | DP on ImageNet-100 · ImageNet-100 | Real Val. Acc.: 68.04 -> 68.01 | No clear effect |
| Use LDM-1.5 rather than LDM-2.1 as the synthetic data source | ViT-B synthetic-data training · ImageNet-100 | Real Val. Acc.: 55.92 -> 59.24 | Helped |
| Use LDM-1.5 rather than LDM-XL as the synthetic data source | ViT-B synthetic-data training · ImageNet-100 | Real Val. Acc.: 52.8 -> 59.24 | Helped |
| Generate samples dynamically without entropy guidance rather than generating the full dataset in advance (ablation) | Synthetic-data training on ImageNet-100 · ImageNet-100 | validation accuracy | No clear effect |
Surprising findings
- Entropy-guided sampling on hard examples causes training loss to increase while validation accuracy improves.
- Excessive entropy-based pruning or guidance degrades performance, so more selection is not monotonically better.
Limitations
- Theory is an idealized high-dimensional linear classifier analysis, not the deep network setting.
- Heavy reliance on hyperparameters (patience, omega, lambda) that require tuning; patience had to be incremented manually.
Experimental setup
Datasets: ImageNet-100 ImageNet-1k
Benchmarks: ImageNet-100 ImageNet-1k ImageNet-V2 ImageNet-Sketch ImageNet-R ImageNet-A
Metrics: top-1 accuracy
Baselines: Static synthetic data (ImageNet-100, ViT-B) Synthetic ImageNet-1k prior work (13M static) Real ImageNet training Oversampling then pruning
Builds on: DDIM classifier-free guidance Static synthetic data scaling (Fan et al.) WSD learning rate schedule
Closest papers by meaning
- Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory (ICLR 2025 · similarity 0.89)
- Scaling Laws of Synthetic Images for Model Training ... for Now (CVPR 2024 · similarity 0.88)
- Real-Fake: Effective Training Data Synthesis Through Distribution Matching (ICLR 2024 · similarity 0.88)
- The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better (NeurIPS 2024 · similarity 0.88)
- Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.88)
- On the Stability of Iterative Retraining of Generative Models on their own Data (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.