Research map / Label-efficient and robust learning / Knowledge distillation
A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize the student with a teacher's FSP matrices and train for fewer iterations | ResNet-26 student · CIFAR-10 | average recognition rate (%): 91.72 -> 92.14 | Helped |
| Initialize the student with a teacher's FSP matrices and train for fewer iterations | ResNet-32 student · CIFAR-100 | average recognition rate (%): 61.32 -> 64.65 | Helped |
| Initialize a student with a teacher's FSP matrices | ResNet-8 student · CIFAR-10 | recognition rate (%): 87.91 -> 88.70 | Helped |
| Initialize a student with a teacher's FSP matrices | ResNet-14 student · CIFAR-100 | recognition rate (%): 58.65 -> 63.33 | Helped |
| Transfer FSP knowledge from an ImageNet-pretrained teacher to a smaller network | ResNet-20 student · CUB200 | recognition rate (%): 47.73 -> 74.26 | Helped |
| Add randomly shuffled FSP matrices to produce distinct student initializations (ablation) | ResNet-26 student ensemble · CIFAR-10 | ensemble recognition rate (%): 93.26 -> 93.67 | Helped |
| Use one intermediate-layer loss instead of three in FitNet (ablation) | FitNet · CIFAR-10 | average recognition rate (%): 89.09 -> 91.72 | Helped |
| Add FSP-based initialization rather than training the student from scratch | ResNet-26 student · CIFAR-10 | training iterations to saturation: three times faster | Helped |
| Use FSP initialization and shorter training for three student networks | ResNet-26 student ensemble · CIFAR-10 | training time: 1.78 times faster | Helped |
Closest papers by meaning
- Variational Information Distillation for Knowledge Transfer (CVPR 2019 · similarity 0.93)
- Improved Knowledge Distillation via Teacher Assistant (AAAI 2020 · similarity 0.91)
- Data-Distortion Guided Self-Distillation for Deep Neural Networks (AAAI 2019 · similarity 0.90)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.90)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.89)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.