Research map / Label-efficient and robust learning / Knowledge distillation
Large Scale Dataset Distillation with Domain Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D3S, a scalable dataset distillation method that reframes distillation as domain shift, matching multivariate-normal layer statistics of a full dataset and relabeling with teacher ensembles, reaching SOTA on Tiny-ImageNet, ImageNet-1K and ImageNet-21K.
Problem
Dataset distillation methods fail to scale to large high-resolution datasets and larger architectures due to memory demands and difficulty generating informative synthetic sets at larger sizes.
Approach
Frame distillation as domain shift and derive a universal bound on distillation loss; approximate the KL divergence between full and distilled data using multivariate Gaussian statistics of intermediate features from pretrained teachers, with incremental statistic updating over batches, ensemble synthesis across teachers, and knowledge-distillation labeling.
Key results
- On Tiny-ImageNet at 50 IPC, D3S reaches 56.4% with ResNet-18, more than 8 points above the prior best.
- On ImageNet-1K at 10 IPC, D3S reaches 39.1% with ResNet-18 versus 28.4% for SRe2L-style baselines on ResNet-50 setting; up to 17.8% gain over prior SOTA is claimed.
- Ablation: multivariate KL loss with full covariance and KL objective yields 27.96% on ImageNet-1K 10/cls vs ~23.5-24% for BatchNorm-based or diagonal variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use D3S instead of CDA | dataset distillation evaluated on ResNet-18 · Tiny-ImageNet, IPC 50 | accuracy: 48.7 -> 56.4 ± 0.3 | Helped |
| use D3S instead of CDA | dataset distillation evaluated on ResNet-18 · Tiny-ImageNet, IPC 100 | accuracy: 53.2 -> 58.8 ± 0.3 | Helped |
| use D3S instead of CDA | dataset distillation evaluated on ResNet-18 · ImageNet-1K, IPC 50 | accuracy: 53.5 -> 60.2 ± 0.1 | Helped |
| use D3S instead of CDA | dataset distillation evaluated on ResNet-18 · ImageNet-1K, IPC 100 | accuracy: 58.0 -> 63.0 ± 0.2 | Helped |
| use D3S instead of CDA | dataset distillation evaluated on ResNet-18 · ImageNet-1K, IPC 200 | accuracy: 63.3 -> 64.6 ± 0.1 | Helped |
| use D3S instead of CDA | dataset distillation evaluated on ResNet-18 · ImageNet-21K, IPC 10 | accuracy: 22.6 -> 26.9 ± 0.1 | Helped |
| use D3S instead of CDA | dataset distillation evaluated on ResNet-18 · ImageNet-21K, IPC 20 | accuracy: 26.4 -> 28.5 ± 0.1 | Helped |
| use multiple networks for both synthesis and labelling (ablation) | D3S · ImageNet-1K, IPC 10 | accuracy | Helped |
| use multiple networks for synthesis as well as labelling, rather than labelling only (ablation) | D3S · ImageNet-1K, IPC 10 | accuracy | Helped |
| add Incremental Statistic Updating (ablation) | D3S with per-batch model randomization · ImageNet-1K, IPC 10 | accuracy | Helped |
| randomize the synthesis model per batch rather than per iteration (ablation) | D3S with multiple models · ImageNet-1K, IPC 10 | accuracy | Helped |
| use multivariate rather than diagonal covariance (ablation) | D3S with KL loss · ImageNet-1K, IPC 10 | accuracy: 23.47±0.21% -> 27.96±0.37% | Helped |
Surprising findings
- Validation-time training duration strongly affects distillation results; training up to 900 epochs raised ImageNet-1K 10 IPC accuracy from 39.1% to 50.1%.
- Only about 25 labeling epochs are needed for Tiny-ImageNet with less than 1.5% drop, suggesting far fewer labels than standard practice.
Limitations
- Multivariate normal approximation of latent features may be poor for complex datasets such as ImageNet-21K, where margins shrink.
- Validation results are highly sensitive to downstream training duration and labeling epochs, complicating comparisons.
- Storage cost of per-epoch augmentation labels grows with epochs, classes and distilled set size.
Experimental setup
Datasets: Tiny-ImageNet ImageNet-1K ImageNet-21K
Benchmarks: Tiny-ImageNet ImageNet-1K ImageNet-21K
Metrics: top-1 accuracy
Baselines: SRe2L CDA
Builds on: SRe2L CDA Nguyen et al. 2021a domain adaptation bound
Closest papers by meaning
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.91)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.91)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.90)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.90)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.90)
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.