Research map / Label-efficient and robust learning / Self-supervised and contrastive
Dataset Distillation for Pre-Trained Self-Supervised Vision Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Linear Gradient Matching, which distills one synthetic image per class so that linear probes on frozen pre-trained self-supervised backbones match real-data gradients, with pyramid parameterization and augmentations enabling cross-model transfer.
Problem
Dataset distillation has focused on training randomly initialized models from scratch, whereas modern vision increasingly uses large pre-trained self-supervised backbones with linear probes; the task of distilling data for optimal linear probing on such features is unexplored.
Approach
Optimize synthetic images so that, after passing through a frozen pre-trained feature extractor, the gradients of the cross-entropy loss with respect to a randomly sampled linear classifier match those from real data (cosine meta-loss), with a multi-scale pyramid image parameterization, decorrelated color space, and differentiable augmentations to reduce overfitting and enable cross-model transfer
Key results
- On ImageNet-1k with one image per class, DINO-v2 linear probe trained on distilled data reaches 75.0% vs 59.7% for best real baseline (Centroids averaged) and 83.0% on full data (DINO-v2 full-dataset result reported as 83.0%).
- Cross-model transfer: CLIP-distilled data gets 84.9% on ImageNet-100 with CLIP, and DINO-distilled data transfers well; cross-model accuracy correlates with mutual k-NN model alignment.
- On Spawrious the method no longer beats real-image baselines on average (48.3% vs 48.4% Centroids), while on fine-grained CUB-2011 it reaches 66.2% average vs 55.7% for Centroids.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use distilled images instead of real-image selection baselines | Linear Gradient Matching · ImageNet-100 and ImageNet-1k | accuracy | Helped |
| Use distilled images instead of real-image selection baselines | Linear Gradient Matching · Stanford Dogs and CUB-200-2011 | accuracy | Helped |
| Use distilled images instead of real-image selection baselines | Linear Gradient Matching · Spawrious | accuracy | Mixed |
| Add color decorrelation during image optimization (ablation) | Linear Gradient Matching · ImageNet-100 cross-model evaluation | accuracy | Helped |
| Add pyramid representation instead of naive pixel optimization (ablation) | Linear Gradient Matching · ImageNet-100 cross-model evaluation | accuracy | Helped |
| Add differentiable augmentations during distillation (ablation) | Linear Gradient Matching · ImageNet-100 | accuracy | Helped |
| Increase augmentations per batch from 1 to 5 and then 10 (ablation) | Linear Gradient Matching with DINO-v2 · ImageNet-100, same-model and cross-model evaluation | accuracy | Helped |
Surprising findings
- DINO-v1 trained only on ImageNet can distill out-of-distribution ArtBench art-style images into colorful synthetic images unlike its nearest real neighbors.
- Cross-model transfer of distilled data is strongly predicted by model alignment, with CLIP-to-MoCo transfer being a notable outlier.
Limitations
- Memory and data-loading bottlenecks limit augmentations per batch (3 for ImageNet-1k) and require loading thousands of real images per step.
- Bi-level setup forces use of nn.DataParallel instead of DistributedDataParallel, causing slowdowns with large batches.
- Does not beat real-image baselines on the Spawrious spurious-correlation dataset.
Experimental setup
Datasets: ImageNet-1k ImageNet-100 Spawrious Waterbirds Stanford Dogs CUB-200-2011 ArtBench Flowers-102 Food-101 ImageNet-Fruits
Benchmarks: ImageNet-1k ImageNet-100 Spawrious Waterbirds Stanford Dogs CUB-200-2011 Flowers-102 Food-101 ArtBench
Metrics: top-1 test accuracy normalized 1-nearest-neighbor accuracy mutual k-NN model alignment
Baselines: Neighbors Centroids Random Full Dataset
Builds on: gradient matching trajectory matching differentiable siamese augmentation feature visualization decorrelated color space Platonic Representation Hypothesis
Closest papers by meaning
- Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.93)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.92)
- Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models (NeurIPS 2026 · similarity 0.90)
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.89)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.89)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.