AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Dataset Distillation for Memorized Data: Soft Labels can Leak Held-Out Teacher Knowledge

Freya Behrens, Lenka Zdeborová

ICLR 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 5 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase fraction rho of teacher data seen by GPT-2 student (ablation)GPT-2 student · random token sequencesheld-out memorized accuracy: 0.213 -> 0.652Improved
raise softmax temperature of soft labels (0.1 to 10) (ablation)transformer student, modular addition · modular addition p=113leakage of held-out memorized dataImproved
use low temperature tau=0.1 soft labels (ablation)transformer student, modular addition · modular addition p=113student behaviorHurt
distill from non-generalizing teacher at tau=10transformer student · modular addition p=113validation accuracyHurt
increase sample complexity alpha beyond identifiability threshold (ablation)logistic regression student · random data d=1600held-out accuracy: 0.55 -> 0.99Improved
shuffle soft labels within class (ablation)logistic regression student · random dataheld-out memorized accuracyHurt
zero out smallest k soft-label entries (ablation)ReLU MLP student · random data c=100, rho=0.7held-out memorized accuracyHurt
zero out one class entry in other classes' soft labels (ablation)ReLU MLP student · random data c=100accuracy on removed classHurt
remove one class from student training inputs but keep soft labels (ablation)ReLU MLP student · random data c=100held-out accuracy on removed classNo clear effect
increase student hidden size (ablation)ReLU MLP student · random data, rho=0.7student training accuracyImproved
increase number of classes clogistic regression · random dataidentifiability thresholdImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy