Research map / Label-efficient and robust learning / Knowledge distillation
Dataset Distillation for Memorized Data: Soft Labels can Leak Held-Out Teacher Knowledge
Freya Behrens, Lenka Zdeborová
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase fraction rho of teacher data seen by GPT-2 student (ablation) | GPT-2 student · random token sequences | held-out memorized accuracy: 0.213 -> 0.652 | Improved |
| raise softmax temperature of soft labels (0.1 to 10) (ablation) | transformer student, modular addition · modular addition p=113 | leakage of held-out memorized data | Improved |
| use low temperature tau=0.1 soft labels (ablation) | transformer student, modular addition · modular addition p=113 | student behavior | Hurt |
| distill from non-generalizing teacher at tau=10 | transformer student · modular addition p=113 | validation accuracy | Hurt |
| increase sample complexity alpha beyond identifiability threshold (ablation) | logistic regression student · random data d=1600 | held-out accuracy: 0.55 -> 0.99 | Improved |
| shuffle soft labels within class (ablation) | logistic regression student · random data | held-out memorized accuracy | Hurt |
| zero out smallest k soft-label entries (ablation) | ReLU MLP student · random data c=100, rho=0.7 | held-out memorized accuracy | Hurt |
| zero out one class entry in other classes' soft labels (ablation) | ReLU MLP student · random data c=100 | accuracy on removed class | Hurt |
| remove one class from student training inputs but keep soft labels (ablation) | ReLU MLP student · random data c=100 | held-out accuracy on removed class | No clear effect |
| increase student hidden size (ablation) | ReLU MLP student · random data, rho=0.7 | student training accuracy | Improved |
| increase number of classes c | logistic regression · random data | identifiability threshold | Improved |
Closest papers by meaning
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.90)
- Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (NeurIPS 2020 · similarity 0.89)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.88)
- Logit Standardization in Knowledge Distillation (CVPR 2024 · similarity 0.87)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.87)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
