Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
Byeongho Heo, Minsik Lee, Sangdoo Yun, Jin Young Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer hidden-neuron activation boundaries with the proposed loss instead of conventional response-transfer methods | WRN16-2 with WRN22-4 teacher · CIFAR-10 test, 1+1 epochs | error rate: 43.37% -> 16.39% | Improved |
| transfer hidden-neuron activation boundaries with the proposed loss | WRN16-2 with WRN22-4 teacher · CIFAR-10, training data 0.1% | error rate: 67.54% -> 50.32% | Improved |
| transfer activation boundaries with the proposed loss rather than the strongest listed baseline | student networks compressed from larger WRNs · CIFAR-10 tiny network | error rate: 28.70% -> 23.27% | Improved |
| use activation-boundary transfer with DTL instead of FITNET + DTL | MobileNet student with ResNet50 teacher · MIT scenes, 5 examples per class | accuracy: 29.55% -> 43.36% | Improved |
| use activation-boundary transfer with DTL instead of FITNET + DTL | MobileNet student with ResNet50 teacher · CUB 2011, 1 example per class | accuracy: 10.70% -> 13.38% | Improved |
| transfer activation boundaries rather than neuron response magnitudes using l2 loss (ablation) | WRN16-4 teacher and student · CIFAR-10 classification | error rate: 10.1% -> 5.1% | Improved |
| transfer activation boundaries rather than neuron response magnitudes using l2 loss (ablation) | WRN16-4 teacher and student · CIFAR-10, layer 1 activation similarity | same activation: 56.1% -> 96.3% | Improved |
| set margin μ to 1 instead of 0.75 (ablation) | Proposed activation-boundary transfer · CIFAR-10, 10% training data | error rate: 13.9% -> 13.1% | Improved |
| set margin μ to 1.5 instead of 1 (ablation) | Proposed activation-boundary transfer · CIFAR-10, 10% training data | error rate: 13.1% -> 12.1% | Improved |
| increase margin μ from 1.5 to 2 (ablation) | Proposed activation-boundary transfer · CIFAR-10, 10% training data | error rate: 12.1% -> 12.1% | No clear effect |
| increase margin μ from 2 to 4 (ablation) | Proposed activation-boundary transfer · CIFAR-10, 10% training data | error rate: 12.1% -> 11.7% | Improved |
| transfer responses from multiple layers rather than layer 3 alone (ablation) | Proposed + KD · CIFAR-10, 10% training data | error rate: 17.23% -> 13.09% | Improved |
Closest papers by meaning
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.87)
- Similarity-Preserving Knowledge Distillation (ICCV 2019 · similarity 0.87)
- Variational Information Distillation for Knowledge Transfer (CVPR 2019 · similarity 0.86)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.84)
- Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks (CVPR 2025 · similarity 0.84)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
