Research map / Label-efficient and robust learning / Self-supervised and contrastive
Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends KRR-ST self-supervised dataset distillation by parameterizing images and representations into PCA-initialized bases with coefficients, predefining rotation augmentations, and using lightweight MLP approximation networks to predict augmented-view representation shifts.
Problem
Self-supervised dataset distillation (distilling images with their self-supervised representations) is storage-heavy, and prior work (KRR-ST) stores each distilled image and representation independently and must drop augmentations due to gradient bias.
Approach
Parameterize distilled images and representations as linear combinations of separate low-dimensional PCA-initialized bases; predefine differentiable augmentations (rotations) and store their representations via lightweight approximation MLPs that predict representation shifts from the unaugmented coefficients.
Key results
- On CIFAR100 with budget N=100 (CIFAR100 target), the method reaches 52.41% linear evaluation accuracy versus 47.00% for KRR-ST.
- On CIFAR10 target with ConvNet, accuracy rises from 72.14% (KRR-ST) to 76.83%; on Stanford Dogs target 15.34% vs 13.42%.
- Ablation: parameterization alone gives 48.57%; adding predefined augmentation and approximation networks gives 52.41%; random basis/coefficient init drops to 22.05%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add image and representation parameterization to KRR-ST (ablation) | KRR-ST · CIFAR100 (N=100) | accuracy: 47.00 $\pm$0.51 -> 48.57 $\pm$0.18 | Helped |
| add predefined augmentation and feature approximation to parameterization (ablation) | Ours · CIFAR100 (N=100) | accuracy: 48.57 $\pm$0.18 -> 52.41 $\pm$0.10 | Helped |
| initialize image and representation bases with principal components and coefficients with projections instead of random values (ablation) | Ours · CIFAR100 (N=100) | accuracy: 22.05 $\pm$3.04 -> 52.41 $\pm$0.10 | Helped |
| use source images to initialize bases and random values to initialize coefficients instead of PC bases and projected coefficients (ablation) | Ours · CIFAR100 (N=100) | accuracy: 52.41 $\pm$0.10 -> 30.99 $\pm$0.25 | Hurt |
| predict augmented representation shifts with approximation networks rather than treating augmented representations as identical (ablation) | Ours · CIFAR100 (N=100) | accuracy: 50.10 -> 52.41 | Helped |
| predict augmented representation shifts with approximation networks rather than a fixed augmentation-specific bias (ablation) | Ours · CIFAR100 (N=100) | accuracy: 50.32 -> 52.41 | Helped |
| use hidden width 8 rather than hidden width 4 in the approximation network (ablation) | Ours · CIFAR100 (N=100) | accuracy: 52.41 -> 51.19 | Hurt |
| use rotation augmentation instead of no augmentation (ablation) | Ours · CIFAR100 (N=100) | accuracy: 48.57 $\pm$0.18 -> 52.41 $\pm$0.10 | Helped |
| use jigsaw augmentation instead of no augmentation (ablation) | Ours · CIFAR100 (N=100) | accuracy: 48.57 $\pm$0.18 -> 50.93 $\pm$0.21 | Helped |
| use crop augmentation instead of no augmentation (ablation) | Ours · CIFAR100 (N=100) | accuracy: 48.57 $\pm$0.18 -> 51.03 $\pm$0.40 | Helped |
| distill with Ours instead of KRR-ST | CIFAR100 source dataset · CUB2011 (ImageNet source) | accuracy: 12.32 -> 12.08 | Hurt |
| distill with Ours instead of KRR-ST | TinyImageNet source dataset · CUB2011 | accuracy: 11.59 $\pm$0.35 -> 11.11 $\pm$0.26 | Hurt |
Surprising findings
- Initializing bases with random Gaussian values yields only 22.05% accuracy, while random real images as bases yield 30.99%, showing initialization is critical.
- Supervised distillation methods (DM, DATM, IDM) often underperform a simple random real-sample baseline on cross-architecture linear evaluation.
Limitations
- Requires about 1.5x more GPU memory than KRR-ST during distillation.
- Only rotation augmentation used in main experiments; other augmentations (jigsaw, crop) yield smaller gains, and the method is limited to differentiable predefined augmentations.
Experimental setup
Datasets: CIFAR100 TinyImageNet ImageNet CIFAR10 CUB2011 Stanford Dogs
Benchmarks: CIFAR100 CIFAR10 CUB2011 Stanford Dogs TinyImageNet ImageNet
Metrics: linear evaluation accuracy standard deviation over three runs
Baselines: Random KMeans DSA DM IDM DATM KRR-ST
Builds on: KRR-ST FRePo MTT Barlow Twins
Closest papers by meaning
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.92)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.91)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.90)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.90)
- Enhancing Dataset Distillation via Non-Critical Region Refinement (CVPR 2025 · similarity 0.89)
- Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.