Research map / Label-efficient and robust learning / Knowledge distillation

DD-Ranking: Rethinking the Evaluation of Dataset Distillation

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Self and data-free distillation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DD-Ranking, a unified evaluation framework for dataset distillation with label-robust (HLR/IOR/LRS) and augmentation-robust (ARS) scores that discount soft-label and augmentation gains by comparing against random selection under identical settings.

Problem

Test accuracy of models trained on distilled datasets is an unreliable metric for comparing dataset distillation methods, because soft labels and stronger augmentation/training recipes inflate results and even random real images can match or beat DD methods under such settings.

Approach

Calibrate each DD method against random selection under the same evaluation recipe (IOR), combine it with the gap to full-data hard-label accuracy (HLR) into a label-robust score, and separately measure augmentation robustness by comparing gains with and without augmentation (ARS).

Key results

• On ImageNet-1K IPC10, most decoupled soft-label DD methods have IOR near or below zero (e.g., SRe2L -15.6, EDC -0.4), so they fail to beat random selection under the same soft-label recipe.
• On CIFAR-10 IPC1, DATM achieves the top LRS of 24.6, versus 14.3 for SRe2L and 16.2 for RDED.
• Removing data augmentation from RDED on ImageNet-1K IPC10 drops its accuracy-based gap; random soft-labeled real images outperform several DD methods on TinyImageNet and ImageNet-1K.

Surprising findings

• Randomly selected real images with soft labels outperform many dataset distillation methods on TinyImageNet and ImageNet-1K.
• Random noise labeled with soft labels achieves non-negligible accuracy, well above chance.

Limitations

• Only supports image classification dataset distillation
• Benchmark covers a subset of existing DD methods, acknowledged as incomplete
• Metric depends on weight lambda and gamma choices and on baseline re-implementation fidelity

Experimental setup

Datasets: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-1K

Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-1K

Metrics: HLR IOR LRS ARS test accuracy

Baselines: DC DSA MTT DM DataDAM DATM SRe2L RDED D4M EDF DWA CDA

Builds on: DC DSA MTT DM SRe2L RDED DATM

Closest papers by meaning

• Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.92)
• A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.89)
• Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.89)
• D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.87)
• Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator (ICLR 2025 · similarity 0.87)
• Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.87)