Research map / Label-efficient and robust learning / Knowledge distillation
Condensed Data Expansion Using Model Inversion for Knowledge Distillation
Kuluhan Binici, Shivam Aggarwal, Cihan Acar, Nam Trung Pham, Karianto Leman, Gim Hee Lee, Tulika Mitra
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| guide model inversion with condensed samples via feature discriminator | Fast · CIFAR-100 (WRN-40-2 -> MobileNet-v2) | student accuracy: 48.21 -> 60.86 | Improved |
| guide model inversion with condensed samples via feature discriminator | CMI · CIFAR-100 (WRN-40-2 -> MobileNet-v2) | student accuracy: 59.04 -> 68.05 | Improved |
| guide model inversion with condensed samples via feature discriminator | CMI · CIFAR-10 (ResNet-34 -> ResNet-18) | student accuracy: 94.84 -> 94.97 | No clear effect |
| guide model inversion with condensed samples via feature discriminator | PRE-DFKD · CIFAR-100 (ResNet-34 -> MobileNet-v2) | student accuracy: 66.56 -> 67.39 | Improved |
| naively add condensed samples to MI synthetic set (no feature alignment) (ablation) | Fast · CIFAR-10 (ResNet-34 -> ResNet-18) | student accuracy: 92.62 -> 92.72 | No clear effect |
| naively add condensed samples to MI synthetic set (no feature alignment) (ablation) | CMI · CIFAR-100 (ResNet-34 -> ResNet-18) | student accuracy: 77.04 -> 77.04 | No clear effect |
| naively add condensed samples to MI synthetic set (no feature alignment) (ablation) | Fast · CIFAR-100 (ResNet-34 -> MobileNet-v2) | student accuracy: 54.62 -> 56.57 | Improved |
| replace condensation method (DSA/DM/MTT) (ablation) | Fast* (ours) · CIFAR-10 (ResNet-34 -> MobileNet-v2) | student accuracy: 88.05 / 87.82 / 87.11 vs naive 86.37 / 86.54 / 86.78 | Improved |
| use real few-shot samples instead of condensed ones (ablation) | PRE-DFKD* · CIFAR-100 (ResNet-34 -> MobileNet-v2) | student accuracy: 67.39 -> 71.63 | Improved |
| use real few-shot samples instead of condensed ones (ablation) | PRE-DFKD* · CIFAR-10 (ResNet-34 -> MobileNet-v2) | student accuracy: 86.89 -> 86.77 | No clear effect |
| replace few-shot KD with expansion via model inversion | PRE-DFKD* vs FSKD · CIFAR-10 (75% pruned VGG-11) | student accuracy: 36.00 -> 68.48 | Improved |
| replace few-shot KD with expansion via model inversion | PRE-DFKD* vs NetGraft · CIFAR-100 (VGG-16 -> ResNet-18, 1 spc) | student accuracy: 55.51 -> 68.54 | Improved |
Closest papers by meaning
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.89)
- What is Dataset Distillation Learning? (ICML 2024 · similarity 0.89)
- What Makes a Good Dataset for Knowledge Distillation? (CVPR 2025 · similarity 0.88)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.88)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.88)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
