Research map / Image generation / Autoregressive and latent generation
The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares finetuning CLIP on targeted synthetic images from Stable Diffusion 1.5 against targeted real images retrieved from its LAION-2B training data, finding retrieval matches or outperforms synthetic data across five classification tasks.
Problem
Whether synthetic images from a text-to-image generator add useful information beyond directly retrieving relevant real images from the generator's upstream training data, in the context of adapting vision models to downstream classification tasks.
Approach
Introduces targeted real-image retrieval from the generator's upstream dataset (LAION-2B), via substring matching and CLIP k-NN search, as a controlled baseline for synthetic training data generated by SynCLR-style prompting of Stable Diffusion; shows retrieval consistently matches or beats synthetic data and analyzes generator artifacts and semantic errors.
Key results
- On FGVC-Aircraft, 139K retrieved LAION-2B images raise linear probing accuracy from 64.9% to 82.7%, versus 68.7% with 139K synthetic images.
- Synthetic data can hurt zero-shot accuracy on ImageNet, StanfordCars, and Flowers102, while retrieved data improves or does not hurt in all settings.
- Matching 15K retrieved Aircraft images requires roughly 500K synthetic images (~33x more data).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace 139K targeted synthetic images with 139K LAION-2B retrieved images | Finetuning LAION-2B pretrained CLIP ViT-B/16 · FGVC-Aircraft | linear probing accuracy: 68.7% -> 82.7% | Helped |
| Finetune on 139K targeted Stable Diffusion images instead of leaving the model unadapted | LAION-2B pretrained CLIP ViT-B/16 · FGVC-Aircraft | linear probing accuracy: 64.9% -> 68.7% | Helped |
| Finetune on 110K targeted Stable Diffusion images instead of leaving the model unadapted | LAION-2B pretrained CLIP ViT-B/16 · DTD | zero-shot accuracy: 56.3% -> 59.6% | Helped |
| Use targeted synthetic images rather than targeted retrieved real images | Finetuned CLIP · ImageNet, StanfordCars, and Flowers102 | zero-shot accuracy | Hurt |
| Scale synthetic adaptation data from 139K to 250K images | Finetuned CLIP · FGVC-Aircraft | linear probing accuracy: 68.7% -> 70.7% | Helped |
| Scale targeted synthetic data beyond 110K images | Finetuned CLIP · DTD | performance | Hurt |
| Scale synthetic adaptation data | Finetuned CLIP · ImageNet, Flowers102, and StanfordCars | zero-shot accuracy | Hurt |
| Add Stable Diffusion generator artifacts to retrieved real images using SDEdit (ablation) | Finetuning CLIP on retrieved images · FGVC-Aircraft | downstream accuracy | Hurt |
| Add Stable Diffusion generator artifacts to retrieved real images using SDEdit (ablation) | Finetuning CLIP on retrieved images · ImageNet | downstream performance | Hurt |
| Use SD v1.1 or SD v1.3 instead of SD v1.5 to generate targeted synthetic data (ablation) | Finetuned CLIP · ImageNet and FGVC-Aircraft | accuracy | Hurt |
| Mix synthetic images into the 139K retrieved Aircraft adaptation set (ablation) | Finetuned CLIP · FGVC-Aircraft | downstream task accuracy | Hurt |
| Use BLIP-2 captions or LAION alt-text instead of LLM-generated prompts for synthetic images (ablation) | Stable Diffusion synthetic data generation · ImageNet | performance | Hurt |
Surprising findings
- Synthetic images receive higher CLIP similarity scores than retrieved images on average, yet still underperform, so CLIP score does not explain the gap.
- Scaling synthetic data often hurts zero-shot accuracy on ImageNet, StanfordCars, and Flowers102, and DTD shows U-shaped scaling.
Limitations
- Only CLIP ViT-B/16 with full finetuning was tested; other backbones and adaptation methods were not evaluated.
- Stable Diffusion is the only text-to-image model with public training data to retrieve from, so other generators were not studied.
- Only model accuracy is studied; robustness and fairness are left for future work.
Experimental setup
Datasets: LAION-2B ImageNet-1K DTD FGVC-Aircraft StanfordCars Flowers102
Benchmarks: ImageNet-1K DTD FGVC-Aircraft StanfordCars Flowers102
Metrics: zero-shot accuracy linear probing accuracy
Baselines: SynCLR off-the-shelf CLIP LAION-2B untargeted training SD v1.1 synthetic data SD v1.3 synthetic data
Builds on: SynCLR SDEdit WiSE-FT CLIP
Closest papers by meaning
- On the Stability of Iterative Retraining of Generative Models on their own Data (ICLR 2024 · similarity 0.90)
- Scaling Laws of Synthetic Images for Model Training ... for Now (CVPR 2024 · similarity 0.90)
- Real-Fake: Effective Training Data Synthesis Through Distribution Matching (ICLR 2024 · similarity 0.89)
- Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.89)
- Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.89)
- Improving the Scaling Laws of Synthetic Data with Deliberate Practice (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.