Research map / Label-efficient and robust learning / Knowledge distillation
Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that within-class similarity in SRe2L-style condensed ImageNet data forces large soft-label storage, and proposes LPLD, which re-batches images within classes with class-wise BN supervision so that simple random soft-label pruning works (40x label compression on ImageNet-1K).
Problem
Large-scale dataset distillation (ImageNet-1K/21K) requires pre-generated soft labels and augmentation records whose storage exceeds the condensed images by ~30x, making relabel storage the bottleneck.
Approach
Batch synthetic images within classes instead of across classes and use class-wise BN statistics as supervision, which increases within-class diversity; this makes simple random label pruning effective and lets soft-label storage shrink while accuracy is maintained or improved.
Key results
- On ImageNet-1K IPC200 with ResNet-18, label storage reduced from 113 GB to 2.8 GB (40x) while reaching 59.6% vs SRe2L's 51.9% at 40x and 57.0% unpruned.
- At 40x label pruning on ImageNet-21K-P IPC20, the method reaches 29.4% vs SRe2L reference, with label storage cut from 1285 GB to 32 GB (8.9% accuracy gain over SRe2L).
- Random label pruning on the improved pool matches carefully selected pruning metrics (e.g., 20.2% random vs ~19-20% for metric-based pruning at 40x on IPC10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LPLD instead of CDA | ImageNet-1K IPC10 label pruning · ImageNet-1K IPC10, 40× | accuracy: 13.2 -> 20.2 | Helped |
| Use LPLD instead of CDA | ImageNet-1K IPC200 label pruning · ImageNet-1K IPC200, 1× | accuracy: 63.3 -> 62.6 | Hurt |
| Add class-wise matching (ablation) | LPLD · ImageNet-1K IPC50, 1× | accuracy: 52.0 -> 54.7 | Helped |
| Add class-wise matching (ablation) | LPLD · ImageNet-1K IPC50, 100× | accuracy: 25.4 -> 22.6 | Hurt |
| Add class-wise supervision (ablation) | LPLD with class-wise matching · ImageNet-1K IPC50, 30× | accuracy: 42.9 -> 45.7 | Helped |
| Add the improved label pool (ablation) | LPLD with class-wise matching and class-wise supervision · ImageNet-1K IPC50, 30× | accuracy: 45.7 -> 48.6 | Helped |
| Add the improved label pool (ablation) | LPLD with class-wise matching · ImageNet-1K IPC50, 30× | accuracy: 42.9 -> 46.2 | Helped |
| Use random pruning instead of metric-based pruning | LPLD · ImageNet-1K IPC10, 40× | accuracy: 19.2 -> 20.2 | Helped |
| Use the improved label pool instead of G-VBSM | ImageNet-1K IPC10 label pruning · ImageNet-1K IPC10, 40× | accuracy: 18.8 -> 20.2 | Helped |
| Use LPLD instead of SRe²L | ImageNet-21K-P IPC20 label pruning · ImageNet-21K-P IPC20, 1× | accuracy: 20.5 -> 30.3 | Helped |
| Use LPLD instead of CDA | Tiny-ImageNet IPC50 label pruning · Tiny-ImageNet IPC50, 1× | accuracy: 48.7 -> 48.8 | No clear effect |
Surprising findings
- Carefully rule-based label pruning metrics are no better than simple random pruning once within-class diversity is improved.
- Scaling to larger networks (ResNet-101) does not necessarily improve pruned accuracy over ResNet-50.
Limitations
- Relies on a single pretrained relabel model; ensemble improvements are shown only as add-ons.
- Performance degrades at very large pruning ratios (e.g., 100x) for small IPCs.
- Theoretical bound on BN update counts relies on approximations (binomial sampling, small-momentum approximation).
Experimental setup
Datasets: ImageNet-1K Tiny-ImageNet ImageNet-21K-P
Benchmarks: Tiny-ImageNet ImageNet-1K ImageNet-21K-P
Metrics: top-1 accuracy storage (GB) cosine similarity maximum mean discrepancy
Baselines: SRe2L CDA RDED G-VBSM FKD FerKD
Builds on: SRe2L CDA DeepInversion FKD RDED
Closest papers by meaning
- GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost (ICLR 2025 · similarity 0.89)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.89)
- Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation (NeurIPS 2025 · similarity 0.87)
- Enhancing Dataset Distillation via Non-Critical Region Refinement (CVPR 2025 · similarity 0.86)
- Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator (ICLR 2025 · similarity 0.86)
- Exploring Learning Complexity for Efficient Downstream Dataset Pruning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.