Research map / Label-efficient and robust learning / Knowledge distillation
Circumventing Outliers of AutoAugment with Knowledge Distillation
Longhui Wei, An Xiao, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Qi Tian
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add knowledge distillation (M=1) (ablation) | RandAugment · CIFAR-100 modified augment space | error rate: 19.5 -> 17.6 | Improved |
| add knowledge distillation (M=7) (ablation) | RandAugment · CIFAR-100 modified augment space | error rate: 55.1 -> 48.0 | Improved |
| add knowledge distillation (M=0, no augmentation) (ablation) | RandAugment · CIFAR-100 modified augment space | error rate: 18.4 -> 18.0 | Improved |
| increase distortion magnitude M from 0 to 7 without KD (ablation) | RandAugment · CIFAR-100 modified augment space | error rate: 18.4 -> 55.1 | Hurt |
| add KD with Wide-ResNet-28-10 teacher to AutoAugment | AutoAugment · CIFAR-100 Wide-ResNet-28-10 | error rate: 17.1 -> 16.2 | Improved |
| add KD with Wide-ResNet-28-10 teacher to AutoAugment | AutoAugment · CIFAR-100 Shake-Shake (26 2x96D) | error rate: 14.3 -> 13.8 | Improved |
| add KD with itself as teacher to AutoAugment | AutoAugment · CIFAR-100 PyramidNet+ShakeDrop | error rate: 10.7 -> 10.6 | Improved |
| add KD with weaker teacher to AutoAugment | AutoAugment · CIFAR-10 Shake-Shake (26 2x96D) | error rate: 2.0 -> 1.8 | Improved |
| add KD to AutoAugment | AutoAugment · CIFAR-10 PyramidNet+ShakeDrop | error rate: 1.5 -> 1.5 | No clear effect |
| add KD during the extra 50 clean-image epochs (ablation) | AutoAugment Wide-ResNet-28-10 · CIFAR-100 | error rate: 16.8 -> 16.2 | Improved |
| use KD (lambda 0.1) on EfficientNet-B0 with M=9 (ablation) | RandAugment · ImageNet | top-1 accuracy: +0.3% | Improved |
| increase K from 5 to larger values (ablation) | KD · ImageNet EfficientNet-B0 | top-1 accuracy | Hurt |
Closest papers by meaning
- Masked Autoencoders Enable Efficient Knowledge Distillers (CVPR 2023 · similarity 0.89)
- Masked Autoencoders Are Stronger Knowledge Distillers (ICCV 2023 · similarity 0.88)
- Structured Knowledge Distillation for Accurate and Efficient Object Detection (TPAMI 2023 · similarity 0.88)
- AutoAugment: Learning Augmentation Strategies From Data (CVPR 2019 · similarity 0.87)
- Knowledge Distillation: A Good Teacher Is Patient and Consistent (CVPR 2022 · similarity 0.87)
- Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation (CVPR 2021 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
