Research map / Label-efficient and robust learning / Knowledge distillation
On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RDED, an optimization-free dataset distillation paradigm that crops high-realism patches from real images with a pre-trained observer model, stitches them into new images, and relabels them with soft labels, achieving fast, diverse, realistic distilled data at high resolution.
Problem
Existing dataset distillation methods are computationally expensive (bi-level optimization), produce non-realistic images that overfit to architectures, and do not scale to large-scale high-resolution datasets such as ImageNet-1K, while coreset methods lack diversity.
Approach
Replace optimization with a two-stage non-optimizing pipeline: select top realism-scored patches (scored by a pre-trained observer model) from a random subset of each class, concatenate N patches into one image to maximize diversity, and relabel with soft labels from the pre-trained teacher; diversity and realism are motivated via V-information.
Key results
- On full ImageNet-1K with IPC=10, RDED reaches 42.3% top-1 with ResNet-18 vs 21.7% for SRe2L, distilling in about 7 minutes on one RTX-4090.
- Per-image synthesis time on ImageNet-1K is 39.89 ms vs 2113.23 ms for SRe2L with ResNet-18 (about 52x faster), with 1.57 GB peak memory vs 9.14 GB.
- Cross-architecture transfer on ImageNet-1K (IPC=10) gives RDED roughly 2x SRe2L accuracy across ResNet-18, EfficientNet-B0 and MobileNet-V2 teachers/students.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RDED instead of SRe2L at IPC=10 | ImageNet-1K distilled data, evaluated with ResNet-18 · ImageNet-1K IPC=10 | top-1 accuracy: 30.9 ± 0.1 -> 42.0 ± 0.1 | Helped |
| Use RDED instead of SRe2L at IPC=50 | ImageNet-1K distilled data, evaluated with ResNet-18 · ImageNet-1K IPC=50 | top-1 accuracy: 60.8 ± 0.5 -> 56.5 ± 0.1 | Hurt |
| Use RDED instead of SRe2L at IPC=10 | ImageNet-100 distilled data, evaluated with ResNet-18 · ImageNet-100 IPC=10 | top-1 accuracy: 11.0 ± 0.2 -> 35.4 ± 0.7 | Helped |
| Use RDED instead of SRe2L at IPC=50 | ImageNet-100 distilled data, evaluated with ResNet-18 · ImageNet-100 IPC=50 | top-1 accuracy: 51.7 ± 1.0 -> 64.8 ± 0.6 | Helped |
| Use RDED instead of SRe2L | Tiny-ImageNet distilled data, evaluated with ResNet-18 at IPC=10 · Tiny-ImageNet IPC=10 | top-1 accuracy: 14.6 ± 1.1 -> 41.9 ± 0.2 | Helped |
| Use RDED instead of SRe2L | CIFAR-10 distilled data, evaluated with ResNet-18 at IPC=10 · CIFAR-10 IPC=10 | top-1 accuracy: 24.3 ± 0.6 -> 37.1 ± 0.3 | Helped |
| Use RDED instead of SRe2L | CIFAR-100 distilled data, evaluated with ResNet-18 at IPC=10 · CIFAR-100 IPC=10 | top-1 accuracy: 30.7 ± 0.3 -> 42.6 ± 0.2 | Helped |
| Use RDED instead of SRe2L | ImageNet-1K distillation with ResNet-18 · ImageNet-1K | synthesis time per image: 2113.23 -> 39.89 | Helped |
| Use RDED instead of SRe2L | ImageNet-1K distillation with ResNet-18 · ImageNet-1K | peak GPU memory: 9.14 -> 1.57 | Helped |
| Use RDED instead of SRe2L for cross-architecture transfer | ImageNet-1K data distilled with ResNet-18, evaluated with EfficientNet-B0 · ImageNet-1K IPC=10 | top-1 accuracy: 11.7 ± 0.2 -> 31.0 ± 0.1 | Helped |
| Use RDED instead of SRe2L for cross-architecture transfer | ImageNet-1K data distilled with ResNet-18, evaluated with MobileNet-V2 · ImageNet-1K IPC=10 | top-1 accuracy: 15.4 ± 0.2 -> 40.4 ± 0.1 | Helped |
| Increase the pre-selected subset size beyond N × IPC (ablation) | RDED · ImageNet-10, ImageNet-100, ImageNet-1K | top-1 accuracy | Mixed |
Surprising findings
- At IPC=1 the method struggles and underperforms optimization-based baselines on several datasets, and it is weaker on small-resolution datasets with few images per class.
- Random uniform pre-selection of a subset before realism-based selection is needed; pure score-based selection across the full class hurts diversity.
Limitations
- Performance drops at IPC=1 relative to optimization-based methods.
- Weaker on small-resolution datasets such as CIFAR-10 at small IPC.
- Relies on a pre-trained observer/teacher model for both patch scoring and relabeling.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K ImageNet-10 ImageNet-100 ImageNette ImageWoof
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K ImageNet-100 ImageNette ImageWoof
Metrics: top-1 validation accuracy synthesis time per image (ms) peak GPU memory (GB)
Baselines: MTT IDM TESLA DATM SRe2L GLaD Herding
Builds on: SRe2L V-information trajectory matching (MTT) GLaD
Closest papers by meaning
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.88)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.87)
- What is Dataset Distillation Learning? (ICML 2024 · similarity 0.86)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.85)
- Distilling Long-tailed Datasets (CVPR 2025 · similarity 0.85)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.