Research map / Label-efficient and robust learning / Knowledge distillation
Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification
Daqian Shi, Xiaolei Diao, Xu Chen, Cédric M John
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add competitive optimization (dynamic best-network teacher) instead of DML | ResNet-56 in ResNet-56/ResNet-152 group · CIFAR100 | Top-1 accuracy: 75.31 -> 76.76 | Improved |
| add competitive distillation instead of independent training | ResNet-56 · CIFAR100 | Top-1 accuracy: 73.76 -> 76.76 | Improved |
| add feature loss L_F (ablation) | competitive optimization · CIFAR100 | Top-1 accuracy: 76.12 -> 76.43 | Improved |
| add stochastic perturbation (ablation) | competitive optimization · CIFAR100 | Top-1 accuracy: 76.12 -> 76.65 | Improved |
| add both L_F and perturbation (ablation) | competitive optimization · CIFAR100 | Top-1 accuracy: 76.12 -> 76.76 | Improved |
| add stochastic perturbation (ablation) | DML · CIFAR100 (ResNet-56) | Top-1 accuracy: 75.31 -> 76.18 | Mixed |
| add feature loss L_F (ablation) | DML · CIFAR100 | Top-1 accuracy: 75.31 -> 75.72 | Improved |
| add perturbation to DML on MobileNet/ResNet-44 group (ablation) | DML · CIFAR100 | Top-1 accuracy: 76.09 -> 75.78 | Hurt |
| add competitive distillation instead of DML | ResNet-56 on ImageNet · ImageNet (ResNet-56/ResNet-152) | Top-1 accuracy: 78.48 -> 78.97 | Improved |
| train 2x ResNet-32 with competitive distillation instead of DML | ResNet-32 · CIFAR100 | Top-1 accuracy: 70.87 -> 71.45 | Improved |
| add competitive distillation instead of DML | ResNet-56 on CIFAR10 with ResNet-56/ResNet-152 · CIFAR10 | Top-1 accuracy: 94.69 -> 94.65 | No clear effect |
| add competitive distillation instead of L-MCL | ResNet-56 · Market-1501 | Rank-1: 90.43 -> 91.28 | Improved |
Closest papers by meaning
- Knowledge Distillation: A Survey (IJCV 2021 · similarity 0.90)
- Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (TPAMI 2021 · similarity 0.89)
- Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (ICCV 2019 · similarity 0.89)
- SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization (NeurIPS 2025 · similarity 0.89)
- A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (CVPR 2017 · similarity 0.89)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
