Research map / Image generation / Fast diffusion sampling
Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Progressive Dataset Distillation (PDD), which distills multiple synthetic subsets across training stages, each conditioned on earlier subsets, and trains on their cumulative union to improve existing dataset distillation methods.
Problem
Dataset distillation methods only match early training dynamics, so a single synthetic subset leaves a substantial performance gap to training on the full dataset.
Approach
Generate a sequence of small synthetic subsets, each distilled from the full-data training dynamics of a later stage starting from the parameters reached after training on earlier subsets, and train progressively on the growing union of subsets; the method wraps existing distillation algorithms (MTT, IDC, DC, DSA) without extra test-time training cost.
Key results
- PDD+MTT improves over MTT on CIFAR-10 IPC-50 from 71.9% to 74.2% and on CIFAR-100 IPC-50 from 47.7% to 52.0%.
- PDD+IDC reaches 76.5% on CIFAR-10 IPC-50 versus 74.5% for IDC.
- Scaling stages reaches about 90% of full-data accuracy on CIFAR-10 with IPC=250 (5% of data).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply progressive distillation to MTT | MTT · CIFAR-10, IPC 10 | test accuracy: 65.3\pm0.7 -> 66.9\pm0.4 | Helped |
| Apply progressive distillation to MTT | MTT · CIFAR-10, IPC 50 | test accuracy: 71.9\pm0.2 -> 74.2\pm0.5 | Helped |
| Apply progressive distillation to IDC | IDC · CIFAR-10, IPC 50 | test accuracy: 74.5\pm0.1 -> 76.5\pm0.4 | Helped |
| Apply progressive distillation to MTT | MTT · CIFAR-100 | test accuracy: 39.6\pm0.3 -> 43.1\pm 0.7; 47.7\pm0.2 -> 52.0\pm0.5 | Helped |
| Apply progressive distillation to MTT | MTT · Tiny-ImageNet | test accuracy: 23.2\pm0.3 -> 27.3\pm0.5; 28.0\pm0.3 -> 29.2\pm0.6 | Helped |
| Train progressively on cumulative unions of synthetic subsets (ablation) | PDD+MTT · CIFAR-10, 5 stages, IPC 10 per stage | test accuracy: 65.9\pm0.4 -> 74.2\pm0.5 | Helped |
| Train progressively on cumulative unions instead of sequentially on each subset (ablation) | PDD+MTT · CIFAR-10, 5 stages, IPC 10 per stage | test accuracy: 67.4\pm0.8 -> 74.2\pm0.5 | Helped |
| Add transitions between distillation stages (ablation) | PDD+MTT · CIFAR-10, 5 stages | test accuracy: 63.6\pm0.7 -> 74.2\pm0.5 | Helped |
| Condition each new synthetic subset on earlier subsets (ablation) | PDD+MTT · CIFAR-10, 5 stages | test accuracy: 71.9\pm0.7 -> 74.2\pm0.5 | Helped |
| Increase the number of stages from 1 to 5 while keeping total IPC at 10 (ablation) | PDD+MTT · CIFAR-10, total IPC 10 | test accuracy: 65.3\pm0.7 -> 66.9\pm0.4 | Helped |
| Increase stages to 10, leaving 1 image per stage (ablation) | PDD+MTT · CIFAR-10, total IPC 10 | test accuracy: 66.9\pm0.4 -> 64.4\pm0.6 | Hurt |
| Distill progressively harder examples using forgetting-score partitions (ablation) | PDD+MTT · CIFAR-10, 3 stages | test accuracy: 71.5\% -> 71.6\% | No clear effect |
Surprising findings
- Union training (training on all subsets at once) with PDD+MTT performs worse than single-stage MTT at P=2 and P=3 on CIFAR-10, because MTT's learned per-set learning rate does not suit a union.
Limitations
- Gains over baselines are modest at some settings (e.g., +0.4% for PDD+IDC on CIFAR-10 IPC-10).
- Hyperparameters of the base methods are reused per stage without further tuning.
- Distillation of IDC on Tiny-ImageNet was deemed too expensive and was not run.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: test accuracy
Baselines: Random selection Herding K-Center Forgetting DC DSA DM CAFE IDC MTT
Builds on: MTT IDC DC DSA forgetting score
Closest papers by meaning
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.91)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.90)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.90)
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.89)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.