Research map / Label-efficient and robust learning / Noisy labels
GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GIFT, a plug-and-play label utilization method for dataset distillation that refines teacher soft labels with smoothed hard labels and trains with a cosine-similarity loss, improving SOTA distilled datasets at no extra cost.
Problem
Models trained on distilled datasets are highly sensitive to the loss function used for soft labels and generalize poorly across optimizers; teacher soft labels are also suboptimal and hard labels are underused.
Approach
Refine each soft label by mixing it with a label-smoothed hard label (gamma=0.1), then train the student with a cosine-similarity loss (motivated by an InfoNCE-based upper bound on V-information and approximate orthogonality of label vectors). It is applied only at evaluation time on existing distilled datasets.
Key results
- On ImageNet-1K IPC=10 cross-optimizer evaluation, GIFT lifts RDED from 0.1% to 26.7% with SGD on ConvNet (the abstract cites a 30.8% gain on RDED cross-optimizer generalization).
- GIFT improves RDED on CIFAR-100 IPC=10 from 47.5% to 50.6% and SRe2L from 33.7% to 38.0% (ConvNet).
- GIFT improves ResNet-18 ImageNet-1K IPC=10 accuracy of RDED by 1.8% (43.2 vs 41.4) and outperforms KD/WSLD/DKD/NKD on RDED synthetic data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GIFT label refinement and cosine similarity loss to SRe²L | SRe²L on CIFAR-100, IPC 10, ConvNet · CIFAR-100 | top-1 accuracy: 33.7 $\pm$ 0.5 -> 38.0 $\pm$ 0.5 | Helped |
| Add GIFT label refinement and cosine similarity loss to RDED | RDED on ImageNet-1K, IPC 10, ConvNet · ImageNet-1K | top-1 accuracy: 20.1 $\pm$ 0.4 -> 24.0 $\pm$ 0.8 | Helped |
| Apply GIFT to RDED-generated data | RDED on Tiny-ImageNet, IPC 50, ResNet-18 · Tiny-ImageNet | top-1 accuracy: 49.5 $\pm$ 0.4 -> 54.5 $\pm$ 0.3 | Helped |
| Add label refinement only (ablation) | RDED on CIFAR-100, IPC 10 · CIFAR-100 | top-1 accuracy: 47.5 -> 48.0 | Helped |
| Add cosine similarity loss only (ablation) | RDED on CIFAR-100, IPC 10 · CIFAR-100 | top-1 accuracy: 47.5 -> 49.5 | Helped |
| Add both label refinement and cosine similarity loss (ablation) | RDED on CIFAR-100, IPC 10 · CIFAR-100 | top-1 accuracy: 47.5 -> 50.6 | Helped |
| Add label refinement only (ablation) | SRe²L on ImageNet-1K, IPC 10 · ImageNet-1K | top-1 accuracy: 12.5 -> 13.3 | Helped |
| Add cosine similarity loss only (ablation) | SRe²L on ImageNet-1K, IPC 10 · ImageNet-1K | top-1 accuracy: 12.5 -> 13.1 | Helped |
| Add both label refinement and cosine similarity loss (ablation) | SRe²L on ImageNet-1K, IPC 10 · ImageNet-1K | top-1 accuracy: 12.5 -> 14.2 | Helped |
| Add label refinement only (ablation) | RDED on ImageNet-1K, IPC 10 · ImageNet-1K | top-1 accuracy: 20.1 -> 21.2 | Helped |
| Add cosine similarity loss only (ablation) | RDED on ImageNet-1K, IPC 10 · ImageNet-1K | top-1 accuracy: 20.1 -> 21.7 | Helped |
| Add both label refinement and cosine similarity loss (ablation) | RDED on ImageNet-1K, IPC 10 · ImageNet-1K | top-1 accuracy: 20.1 -> 24.0 | Helped |
Surprising findings
- Replacing KL divergence with soft cross-entropy for SRe2L drops Tiny-ImageNet accuracy by about 13%, showing high sensitivity to the loss choice.
- Distilled-data models trained with standard losses (e.g., KL, SGD) can collapse to near-zero accuracy under a different optimizer, which GIFT largely fixes.
Limitations
- Theoretical justification relies on approximations (orthogonality of label vectors, approximated denominator) and the bound is loose.
- Only applied to evaluation with existing distilled datasets; does not change the distillation process itself.
- Gamma=0.1 is tuned on the evaluated benchmarks and may not transfer across settings.
Experimental setup
Datasets: CIFAR-100 Tiny-ImageNet ImageNet-1K
Benchmarks: CIFAR-100 Tiny-ImageNet ImageNet-1K
Metrics: top-1 accuracy
Baselines: SRe2L RDED DATM G-VBSM CDA KD WSLD DKD NKD
Builds on: SRe2L RDED DATM G-VBSM InfoNCE
Closest papers by meaning
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.91)
- Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? (NeurIPS 2024 · similarity 0.89)
- Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator (ICLR 2025 · similarity 0.88)
- Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation (NeurIPS 2025 · similarity 0.88)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.87)
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.