Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-Supervised Dataset Distillation for Transfer Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills an unlabeled source dataset into a small synthetic set via MSE-based kernel ridge regression against a self-supervised (Barlow Twins) target model, for transfer learning, architecture generalization, and target data-free KD.
Problem
Dataset distillation methods are designed for supervised training and do not produce distilled sets that transfer well when used for self-supervised pre-training; naive bilevel optimization with SSL losses is unstable because augmentation/masking randomness biases the meta-gradient.
Approach
Replace the SSL inner objective with an MSE regression onto learnable target representations (no randomness), match the inner model's outputs on the full dataset to a frozen Barlow Twins target model, and simplify the inner loop to kernel ridge regression on a pool of feature extractors.
Key results
- On CIFAR100 source (2% compression, ConvNet3), KRR-ST reaches 41.54% on Aircraft and 28.68% on Cars vs. 37.67% and 23.12% for the best baseline KIP.
- On TinyImageNet source (ConvNet4, 2,000 images), KRR-ST reaches 49.26% on Cars and 71.16% on Flowers, versus 30.35% and 66.07% for the best baseline MTT.
- In target data-free knowledge distillation on CIFAR10, KRR-ST achieves 58.31% with ConvNet4, while supervised dataset distillation baselines (DSA, DM, MTT, FRePo) perform worse than Random for several architectures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize target representations from the target model instead of standard normal values (ablation) | KRR-ST with CIFAR100 source · CIFAR10 | accuracy: 87.56 -> 88.87 | Helped |
| Initialize target representations from the target model instead of standard normal values (ablation) | KRR-ST with CIFAR100 source · Aircraft | accuracy: 25.54 -> 32.94 | Helped |
| Initialize target representations from the target model instead of standard normal values (ablation) | KRR-ST with CIFAR100 source · Cars | accuracy: 17.75 -> 23.92 | Helped |
| Initialize target representations from the target model instead of standard normal values (ablation) | KRR-ST with CIFAR100 source · CUB2011 | accuracy: 16.68 -> 24.38 | Helped |
| Initialize target representations from the target model instead of standard normal values (ablation) | KRR-ST with CIFAR100 source · Dogs | accuracy: 20.82 -> 23.39 | Helped |
| Initialize target representations from the target model instead of standard normal values (ablation) | KRR-ST with CIFAR100 source · Flowers | accuracy: 54.94 -> 64.28 | Helped |
| Use KRR-ST distilled CIFAR100 instead of the strongest baseline distillation method | ConvNet3 transfer learning · Aircraft | accuracy: 37.67 -> 41.54 | Helped |
| Use KRR-ST distilled CIFAR100 instead of the strongest baseline distillation method | ConvNet3 transfer learning · Cars | accuracy: 23.12 -> 28.68 | Helped |
| Use KRR-ST distilled TinyImageNet instead of the strongest baseline distillation method | ConvNet4 transfer learning · Cars | accuracy: 30.35 -> 49.26 | Helped |
| Use KRR-ST distilled ImageNet instead of training without pre-training | ConvNet4 transfer learning · Cars | accuracy: 23.62 -> 46.95 | Helped |
| Use KRR-ST distilled TinyImageNet as a surrogate dataset instead of the strongest baseline | Target data-free KD on CIFAR10 · ConvNet4 | accuracy: 52.00 -> 58.31 | Helped |
| Use KRR-ST distilled TinyImageNet as a surrogate dataset instead of the strongest baseline | Target data-free KD on CIFAR10 · VGG11 | accuracy: 53.86 -> 62.15 | Helped |
Surprising findings
- Supervised dataset distillation methods (DSA, DM, MTT, FRePo) sometimes perform worse than random selection in target data-free knowledge distillation.
- Back-propagating through SSL augmentations or masking is empirically unstable, and the paper proves the meta-gradient is biased in that setting.
Limitations
- Baselines such as DSA, DM and MTT could not be run on ImageNet due to large memory consumption, so ImageNet comparisons only include FRePo and Random.
- Only classification is used for fine-tuning in this work.
- Requires a pre-trained self-supervised target model on the full source dataset before distillation.
Experimental setup
Datasets: CIFAR100 TinyImageNet ImageNet ImageNette CIFAR10 Aircraft Stanford Cars CUB2011 Stanford Dogs Flowers
Benchmarks: CIFAR10 Aircraft Stanford Cars CUB2011 Stanford Dogs Flowers CIFAR100 TinyImageNet ImageNet
Metrics: test accuracy FID
Baselines: w/o pre Random Kmeans DSA DM MTT KIP FRePo
Builds on: Kernel Inducing Points FRePo Barlow Twins Dataset Condensation with Gradient Matching Matching Training Trajectories
Closest papers by meaning
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.89)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.88)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.87)
- Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.87)
- Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.