Research map / Label-efficient and robust learning / Knowledge distillation

Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Self and data-free distillation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks seven SOTA dataset distillation methods against three coreset selection strategies (random, AUM, concept-guided AUM) on ImageNet-1K, ImageNet-100, and ImageNette under hard-label, single-teacher, and multi-teacher protocols, finding coresets match or beat DD at lower construction cost.

Problem

Dataset distillation methods are evaluated under inconsistent protocols (ERM vs. soft-label teacher supervision), and claims that DD outperforms coreset selection are made without systematic comparison on large-scale datasets.

Approach

Standardize evaluation across three training protocols (hard label, single-teacher RDED, multi-teacher EDC) and compare SOTA DD methods directly with coreset selection using a class-balanced coverage-centric sampler, also measuring representativeness, diversity, FID, and construction cost.

Key results

• On ImageNet-1K hard-label, AUM with ResNet-18 at 10 IPC reaches 18.9% vs. 15.6% for the best DD method (ManifoldGD), a 3.3-point gain.
• Under single-teacher RDED evaluation at 50 IPC on ImageNet-1K, Concepts ties AUM at 66.0% with ResNet-50, matching or exceeding the best DD method (65.6%).
• AUM construction takes 6.0 hours at both 10 and 50 IPC, while ManifoldGD takes 12.8 and 51.7 hours at 10 and 50 IPC.

Surprising findings

• Many DD methods underperform even a simple random class-balanced subset under all three evaluation protocols on ImageNet-1K.
• On ImageNette with 10 classes, DD methods produce better condensed sets than coresets, reversing the trend seen on larger datasets.

Limitations

• Restricted to image classification and ImageNet-1K and its subsets, since many DD methods release distilled data only for ImageNet-1K.
• Generative-model-based DD methods are impractical to apply to new domains without retraining generative models on the full dataset.
• Generative model and teacher training costs are excluded from the timing analysis.

Experimental setup

Datasets: ImageNet-1K ImageNet-100 ImageNette

Benchmarks: ImageNet-1K ImageNet-100 ImageNette

Metrics: top-1 accuracy representativeness diversity FID construction time (hrs)

Baselines: VLCP D3HR Minimax DiT-Distillation MGD ManifoldGD FADRM+ Random

Builds on: SRe2L RDED EDC CCS (coverage-centric coreset selection) AUM

Closest papers by meaning

• DD-Ranking: Rethinking the Evaluation of Dataset Distillation (NeurIPS 2026 · similarity 0.92)
• Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.91)
• Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.90)
• Enhancing Dataset Distillation via Non-Critical Region Refinement (CVPR 2025 · similarity 0.89)
• A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.89)
• Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator (ICLR 2025 · similarity 0.89)