AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Rethinking Data Distillation: Do Not Overlook Calibration

Dongyao Zhu, Bowen Lei, Jie Zhang, Yanbo Fang, Yiqun Xie, Ruqi Zhang, Dongkuan Xu

ICCV 2023 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 13 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
apply masked temperature scaling instead of standard temperature scalingMTT-distilled CIFAR10 DDNN · CIFAR10ECE (%): 7.45 -> 1.20Improved
apply mixup, label smoothing, or focal loss to DDNNsMTT-distilled CIFAR10 DDNN · CIFAR10ECE (%): 4.93 -> 21.06 (mixup), 25.01 (LS), 6.62 (FL)Hurt
add masked distillation training (10% mask) during distillationMTT · CIFAR10 accuracyaccuracy (%): 70.48 -> 69.98No clear effect
combine MDT with MTS using fixed mask ratio (ablation)MTS · CIFAR10ECE (%): 1.20 -> 2.50Hurt
sample the MDT mask ratio dynamically from 0 to 0.1 combined with MTS (ablation)MDT + MTS · MTT CIFAR10 IPC=10ECE (%): 1.36 -> 1.13Improved
apply MDT alone versus best other method (ablation)MTT · Tiny ImageNetECE (%): 2.42 -> 7.44Hurt
vary validation set size for MTS from 10% to 50% (ablation)MTS · ECE across backbonesECE (%)No clear effect
increase mask ratio r in MTS from 0.1 to 0.5 (ablation)MTS · CIFAR10 and CIFAR100ECE (%)No clear effect
use MTS on IPC=1 distillation without validation dataMTS · CIFAR10 IPC=1ECE (%): 10.15 -> 1.81Improved
hold out 10% of distilled data for temperature scalingMTT ImageNette · ImageNette accuracyaccuracy (%): 63.04 -> 61.36Hurt
apply MTS to full-data networks versus temperature scalingFDNN CIFAR10 · CIFAR10 full dataECE (%): 0.99 -> 2.67No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy