Research map / Label-efficient and robust learning / Knowledge distillation
A Label is Worth A Thousand Images in Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that soft labels, not synthetic image generation, drive most of the performance of state-of-the-art dataset distillation methods, and characterizes the structured information and data-knowledge trade-off in soft labels.
Problem
Dataset distillation methods generate synthetic images with sophisticated techniques, but it is unclear what drives their performance and what makes training data "good" for data-efficient learning.
Approach
A simple baseline that samples random real training images and labels them with soft labels from early-stopped pretrained experts matches SOTA distillation methods, showing soft labels (especially their structured information) are the key factor; the authors also establish a data-knowledge scaling law and Pareto frontier.
Key results
- On ImageNet-1K at IPC=10 with ResNet18 students, the random-image soft label baseline reaches 23.9% accuracy, comparable to SRe2L-style distilled soft labels.
- Soft label baseline with expert-ensemble labels improves TinyImageNet IPC=10 accuracy from 27.2% to 30.1%.
- BPTT-learned labels with frozen random images reach 13.5% at IPC=1 on TinyImageNet, versus 7.6% for a single expert label and 1.6% for hard labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace hard labels with expert-generated soft labels on random training images (ablation) | ImageNet-1K dataset distillation · ImageNet-1K IPC=1 | accuracy: 0.6 ± 0.1 -> 7.3 ± 0.2 | Helped |
| Replace synthetic image generation with random training images while retaining soft labels (ablation) | SRe²L · ImageNet-1K IPC=10 | accuracy: 21.3 ± 0.6 -> 25.8 ± 0.7 | Mixed |
| Use the soft label baseline instead of Ra-BPTT | TinyImageNet distillation · TinyImageNet IPC=1 | accuracy: 20.1 ± 0.3 -> 7.6 ± 0.3 | Hurt |
| Use the soft label baseline instead of Ra-BPTT | CIFAR-100 distillation · CIFAR-100 IPC=10 | accuracy: 47.5 ± 0.2 -> 34.3 ± 0.2 | Hurt |
| Average logits from 10 experts rather than use a single expert (ablation) | Soft label baseline · TinyImageNet IPC=10 | accuracy: 27.2 ± 0.4 -> 30.1 ± 0.2 | Helped |
| Learn labels with BPTT rather than use hard labels (ablation) | Fixed random training images · TinyImageNet IPC=1 | accuracy: 1.6 ± 0.1 -> 13.5 ± 0.3 | Helped |
| Learn labels with MTT rather than use hard labels (ablation) | Fixed random training images · TinyImageNet IPC=10 | accuracy: 7.3 ± 0.2 -> 10.3 ± 0.2 | Helped |
| Increase softmax temperature (ablation) | Expert soft labels · TinyImageNet IPC=10 | accuracy | Helped |
| Swap a top-ranked softmax label with the last-ranked label (ablation) | Expert soft labels · TinyImageNet | accuracy | Hurt |
| Remove all images of a class while retaining that class's soft-label values on other images (ablation) | Soft label baseline · TinyImageNet | accuracy | No clear effect |
| Remove a class's soft-label values from other images while retaining its class images (ablation) | Soft label baseline · TinyImageNet IPC=1 | accuracy: 0% test accuracy for removed classes | Hurt |
| Select easiest training images by cross-entropy instead of random images (ablation) | Soft label baseline · TinyImageNet IPC=1, 10, 50 | accuracy: ≈ 1% | Helped |
Surprising findings
- Hard-label students on images from the removed class reach 0% accuracy, while removing images (keeping soft labels) costs little, so soft labels can enable zero-shot classification of unseen classes.
- Learning labels with the MTT objective fails to meaningfully improve over hard labels, whereas BPTT-learned labels converge to early-stopped expert labels.
Limitations
- Only image classification tasks on small/medium image datasets were evaluated.
- Images were fixed to random real samples; joint image-label synthesis was not studied.
- Advanced methods such as SRe2L still outperform the baseline in some settings, indicating images and labels both matter.
Experimental setup
Datasets: ImageNet-1K downsized ImageNet-1K TinyImageNet CIFAR-100 CIFAR-10
Benchmarks: ImageNet-1K TinyImageNet CIFAR-100 CIFAR-10
Metrics: student test accuracy
Baselines: Ra-BPTT MTT DM SRe2L FRePo TESLA CutMix soft labels hard-label training
Builds on: dataset distillation (bi-level optimization) Matching Training Trajectories (MTT) SRe2L Ra-BPTT knowledge distillation
Closest papers by meaning
- GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost (ICLR 2025 · similarity 0.91)
- DD-Ranking: Rethinking the Evaluation of Dataset Distillation (NeurIPS 2026 · similarity 0.89)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.89)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.89)
- Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? (NeurIPS 2024 · similarity 0.89)
- Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.