Research map / Label-efficient and robust learning / Knowledge distillation
Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MKDT, a dataset distillation method for self-supervised pre-training that matches trajectories of students trained with a KD (MSE-to-teacher-representation) objective, reducing SSL gradient variance, and reports up to 13% higher downstream accuracy than prior SSL distillation.
Problem
Dataset distillation for self-supervised learning pre-training is unsolved because SSL gradients have high variance, which breaks trajectory matching and makes existing supervised DD methods and KRR-ST ineffective.
Approach
Train a ResNet-18 teacher with Barlow Twins, then train small ConvNet students to regress teacher representations with a supervised MSE loss; their lower-variance trajectories serve as expert trajectories for standard MTT-style distillation of synthetic images.
Key results
- On CIFAR10 pre-training with a 2% distilled set and 1% downstream labels, MKDT reaches 44.36% CIFAR10 accuracy vs 37.19% for KRR-ST and 36.35% for random subset.
- MKDT outperforms KRR-ST by up to 13% in downstream accuracy across datasets and label fractions, per the abstract.
- With 5% distilled sets, MKDT reaches 58.37% CIFAR10 accuracy (5% labels) vs 48.13% for random subsets on CIFAR10 pre-training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MKDT instead of KRR-ST for CIFAR10 2% distilled-set pre-training | SSL pre-training on CIFAR10 · CIFAR10; 1% downstream labels | accuracy: 37.19 ± 0.49 -> 44.36 ± 1.61 | Helped |
| Use MKDT instead of the strongest listed subset baseline | SSL pre-training on CIFAR100 · CIFAR100; 1% downstream labels | accuracy: 10.03 ± 0.12 -> 12.36 ± 0.67 | Helped |
| Use MKDT instead of KRR-ST | SSL pre-training on TinyImageNet · Tiny ImageNet; 1% downstream labels | accuracy: 3.32 ± 0.22 -> 3.87 ± 0.05 | Helped |
| Increase the distilled-set size from 2% to 5% | MKDT on CIFAR100 · CIFAR100; 5% downstream labels | accuracy: 22.64 ± 0.42 -> 25.18 ± 0.67 | Helped |
| Initialize synthetic data with high-loss examples rather than random examples (ablation) | MKDT on CIFAR100 · CIFAR100; 1% downstream labels | accuracy: 11.44 ± 0.36 -> 12.36 ± 0.67 | Helped |
| Use a SimCLR-trained teacher instead of a Barlow Twins-trained teacher (ablation) | MKDT on CIFAR100 · CIFAR100; 1% downstream labels | accuracy: 7.35 ± 0.54 -> 9.44 ± 0.37 | Helped |
| Use MKDT rather than random-subset pre-training | CIFAR100 5% distilled set transferred to ResNet-18 · CIFAR100; 5% downstream labels | accuracy: 16.82 ± 0.69 -> 21.51 ± 0.17 | Helped |
Surprising findings
- Naive application of trajectory matching (MTT) to SSL fails, attributed to high variance of SSL gradients.
- KRR-ST does not outperform SSL pre-training on random real images in the fine-tuning setting of its original paper.
- The larger ResNet-18 slightly underperforms the smaller ResNet-10 on downstream tasks, including without pre-training.
Limitations
- Gains on some downstream datasets (e.g., Aircraft, CUB2011, Dogs) are small or within noise.
- Requires training a large teacher with SSL first, which is costly; the distillation itself depends on teacher quality.
- Evaluated only on small-resolution image datasets with ConvNet and ResNet students.
Experimental setup
Datasets: CIFAR10 CIFAR100 TinyImageNet Aircraft CUB2011 Stanford Dogs Flowers
Benchmarks: CIFAR10 CIFAR100 TinyImageNet Aircraft CUB2011 Dogs Flowers
Metrics: linear probe test accuracy downstream accuracy
Baselines: Random Subset SAS Subset KRR-ST High Loss Subset Full Data
Builds on: Matching Training Trajectories (MTT) Barlow Twins KD for SSL (MSE to teacher representations) KRR-ST
Closest papers by meaning
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.91)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.91)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.90)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.89)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.89)
- Distilling Long-tailed Datasets (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.