Research map / Label-efficient and robust learning / Knowledge distillation
What is Dataset Distillation Learning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes what information dataset distillation stores on CIFAR-10, finding distilled data acts like early-stopped real-data training and that individual distilled images carry semantic meaning via influence functions.
Problem
Dataset distillation produces small synthetic datasets that train good classifiers, but little is understood about how the information is stored, how it behaves relative to real data, and whether individual distilled points are meaningful.
Approach
Probes distilled data from three angles: mixing with real data during training, comparing predictions and loss curvature (Hessian trace/eigenvalues) of models trained on distilled data versus early-stopped real-data models, and using leave-one-out influence with LLaVA-derived semantic attributes to interpret individual distilled images.
Key results
- Mixing 10 distilled images per class with 2-50 real images per class on CIFAR-10 can decrease accuracy relative to distilled data alone, unlike the real-data baseline which improves steadily.
- Models trained on distilled data agree more with early-stopped real-data models than with subset-trained models matched in accuracy; BPTT and trajectory matching distilled data reach flat loss-curvature regions within about 300 iterations.
- Influence of individual BPTT distilled images is predictive of semantic attributes (e.g., yellow cars, planes on runways), with some distilled images highly associated with specific semantics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| mix real images into training with distilled images | models trained on distilled data · CIFAR-10 | accuracy | Hurt |
| clip BPTT distilled-image pixel values to between 0 and 1 (ablation) | BPTT · CIFAR-10 | accuracy: 58\% to 44\% | Hurt |
| train on BPTT distilled data for an additional 300 iterations after real-data pretraining (ablation) | model pretrained on real data for 300 iterations · CIFAR-10 | test accuracy: less than 2\% improvement | No clear effect |
| train on trajectory-matching distilled data for an additional 300 iterations after real-data pretraining (ablation) | model pretrained on real data for 300 iterations · CIFAR-10 | test accuracy: less than 2\% improvement | No clear effect |
| train on distribution-matching distilled data for an additional 300 iterations after real-data pretraining (ablation) | model pretrained on real data for 300 iterations · CIFAR-10 | test accuracy: 5-7 \% decrease | Hurt |
| train on gradient-matching distilled data for an additional 300 iterations after real-data pretraining (ablation) | model pretrained on real data for 300 iterations · CIFAR-10 | test accuracy: 5-7 \% decrease | Hurt |
| learn the learning rate during trajectory-matching distillation (ablation) | Trajectory matching · CIFAR-10 | classification accuracy | Helped |
| increase the distilled-data storage budget (ablation) | BPTT and trajectory matching · CIFAR-10 | classification accuracy | Helped |
| use early-stopped real-data training instead of a random real-data subset as the comparison model | models trained on distilled data · CIFAR-10 | prediction agreement | Helped |
| mix real data with distilled data during training | models trained on distilled data · CIFAR-100 | accuracy | Mixed |
| mix real data with distilled data during training | models trained on distilled data · TinyImageNet | accuracy | Mixed |
| mix real data with distilled data during training | models trained with IDC and FreD · CIFAR-10 | accuracy | Mixed |
Surprising findings
- Adding real data to distilled data can decrease accuracy of the trained classifier.
- Clipping out-of-range pixel values of BPTT distilled images reduces retrained accuracy from 58% to 44%.
Limitations
- Analysis is largely restricted to CIFAR-10 with a small ConvNet; generalization is tested on limited additional datasets.
- Semantic attributes were compiled by manual inspection of LLaVA responses, which is subjective.
- Curvature analysis could not draw firm conclusions for distribution and gradient matching because the trace fluctuates greatly.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet
Metrics: classification accuracy prediction agreement Hessian trace Hessian eigenvalue density influence (leave-one-out loss difference) precision-recall AUC
Baselines: BPTT distribution matching gradient matching trajectory matching IDC FreD random real subsets early-stopped models weight decay models
Builds on: BPTT (backpropagation through time dataset distillation) distribution matching gradient matching trajectory matching influence functions Hutchinson trace estimation
Closest papers by meaning
- What Makes a Good Dataset for Knowledge Distillation? (CVPR 2025 · similarity 0.89)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.88)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.88)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.88)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.88)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.