Research map / Label-efficient and robust learning / Knowledge distillation
Dataset Distillation: A Comprehensive Review
Ruonan Yu, Songhua Liu, Xinchao Wang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use trajectory/kernel-based matching (FRePo) instead of gradient matching (DC) | dataset distillation · CIFAR-100 (10 Img/Cls) | test accuracy (%): 25.4 ± 0.3 -> 41.3 ± 0.2 | Improved |
| use multi-step parameter matching (MTT) instead of distribution matching (DM) | dataset distillation · CIFAR-10 (1 Img/Cls) | test accuracy (%): 26.5 ± 0.4 -> 46.3 ± 0.8 | Improved |
| use multi-step parameter matching (MTT) instead of single-step gradient matching with augmentation (DSA) | dataset distillation · Tiny-ImageNet (10 Img/Cls) | test accuracy (%): 16.3 ± 0.2 -> 23.2 ± 0.2 | Improved |
| add momentum to inner-loop parameter update (DD vs DC) | dataset distillation · CIFAR-10 (10 Img/Cls) | test accuracy (%): 44.9 ± 0.5 -> 60.2 ± 0.4 | Improved |
| add momentum in inner loop | DD · MNIST/CIFAR | test accuracy (%) | Improved |
| evaluate DD-distilled data on other architectures (Conv-NN instead of Conv) | DD · CIFAR-10 (10 Img/Cls) | cross-architecture accuracy (%): 60.2 ± 0.4 -> 17.8 ± 2.7 | Hurt |
| evaluate DC-distilled data on Conv-NN instead of Conv | DC · CIFAR-10 (10 Img/Cls) | cross-architecture accuracy (%): 44.9 ± 0.5 -> 31.9 ± 0.6 | Hurt |
| increase images per class from 1 to 50 for MTT | MTT · CIFAR-100 | test accuracy (%): 24.3 ± 0.3 -> 47.7 ± 0.2 | Improved |
| use FRePo instead of MTT at 50 Img/Cls | dataset distillation · CIFAR-100 (50 Img/Cls) | test accuracy (%): 47.7 ± 0.2 -> 44.3 ± 0.2 | Hurt |
Closest papers by meaning
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.92)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.92)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.92)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.92)
- Taming Diffusion for Dataset Distillation with High Representativeness (ICML 2025 · similarity 0.91)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
