Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation with Adversarial Samples Supporting Decision Boundary
Byeongho Heo, Minsik Lee, Sangdoo Yun, Jin Young Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BSS-based boundary loss for knowledge distillation | Student classifiers distilled from ResNet26 · CIFAR-10 | accuracy: 86.66% -> 87.32% | Improved |
| Add BSS distillation to FSP | FSP knowledge distillation · CIFAR-10, ResNet8 student | accuracy: 87.07% -> 87.52% | Improved |
| Add BSS distillation to FSP | FSP knowledge distillation · ImageNet 32×32 | Top5 acc: 57.14% -> 57.27% | Improved |
| Use proposed BSS distillation rather than FITNET+Hinton | ResNet10 student with ResNet42 teacher · TinyImageNet | Top1 acc: 53.52% -> 52.99% | Hurt |
| Use BSS samples rather than FGSM samples (ablation) | Knowledge distillation · CIFAR-10, ResNet8 student | accuracy: 87.06% -> 87.32% | Improved |
| Use BSS samples rather than DeepFool samples (ablation) | Knowledge distillation · CIFAR-10, ResNet8 student | accuracy: 86.95% -> 87.32% | Improved |
| Use BSS samples rather than random noise (ablation) | Knowledge distillation · CIFAR-10, ResNet8 student | accuracy: 86.73% -> 87.32% | Improved |
| Choose base samples with the proposed selection rule rather than using all samples (ablation) | BSS distillation · CIFAR-10, ResNet8 student | accuracy: 87.18% -> 87.32% | Improved |
| Choose base samples with the proposed selection rule rather than randomly (ablation) | BSS distillation · CIFAR-10, ResNet8 student | accuracy: 87.10% -> 87.32% | Improved |
| Sample target classes according to teacher class probabilities rather than randomly (ablation) | BSS distillation · CIFAR-10, ResNet8 student | accuracy: 87.08% -> 87.32% | Improved |
| Use BSS distillation rather than classification-only training | ResNet8 student · CIFAR-10 | accuracy: 86.02% -> 87.32% | Improved |
Closest papers by meaning
- Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples (CVPR 2024 · similarity 0.88)
- Towards Understanding Knowledge Distillation (ICML 2019 · similarity 0.88)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.86)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.86)
- Hybrid Data-Free Knowledge Distillation (AAAI 2025 · similarity 0.85)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
