Research map / Label-efficient and robust learning / Knowledge distillation
Rethinking Data Distillation: Do Not Overlook Calibration
Dongyao Zhu, Bowen Lei, Jie Zhang, Yanbo Fang, Yiqun Xie, Ruqi Zhang, Dongkuan Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply masked temperature scaling instead of standard temperature scaling | MTT-distilled CIFAR10 DDNN · CIFAR10 | ECE (%): 7.45 -> 1.20 | Improved |
| apply mixup, label smoothing, or focal loss to DDNNs | MTT-distilled CIFAR10 DDNN · CIFAR10 | ECE (%): 4.93 -> 21.06 (mixup), 25.01 (LS), 6.62 (FL) | Hurt |
| add masked distillation training (10% mask) during distillation | MTT · CIFAR10 accuracy | accuracy (%): 70.48 -> 69.98 | No clear effect |
| combine MDT with MTS using fixed mask ratio (ablation) | MTS · CIFAR10 | ECE (%): 1.20 -> 2.50 | Hurt |
| sample the MDT mask ratio dynamically from 0 to 0.1 combined with MTS (ablation) | MDT + MTS · MTT CIFAR10 IPC=10 | ECE (%): 1.36 -> 1.13 | Improved |
| apply MDT alone versus best other method (ablation) | MTT · Tiny ImageNet | ECE (%): 2.42 -> 7.44 | Hurt |
| vary validation set size for MTS from 10% to 50% (ablation) | MTS · ECE across backbones | ECE (%) | No clear effect |
| increase mask ratio r in MTS from 0.1 to 0.5 (ablation) | MTS · CIFAR10 and CIFAR100 | ECE (%) | No clear effect |
| use MTS on IPC=1 distillation without validation data | MTS · CIFAR10 IPC=1 | ECE (%): 10.15 -> 1.81 | Improved |
| hold out 10% of distilled data for temperature scaling | MTT ImageNette · ImageNette accuracy | accuracy (%): 63.04 -> 61.36 | Hurt |
| apply MTS to full-data networks versus temperature scaling | FDNN CIFAR10 · CIFAR10 full data | ECE (%): 0.99 -> 2.67 | No clear effect |
Closest papers by meaning
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.87)
- On the Limitations of Temperature Scaling for Distributions with Overlaps (ICLR 2024 · similarity 0.87)
- On Temperature Scaling and Conformal Prediction of Deep Classifiers (ICML 2025 · similarity 0.86)
- Consistency-Guided Temperature Scaling Using Style and Content Information for Out-of-Domain Calibration (AAAI 2024 · similarity 0.86)
- HoTS: Homophily-Aware Temperature Scaling for Graph Neural Network Calibration (NeurIPS 2026 · similarity 0.86)
- A Benchmark Study on Calibration (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
