Research map / Label-efficient and robust learning / Knowledge distillation
Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Angular-KD, which attaches lightweight linear view-augmentation heads to a single teacher and trains them with inter- and intra-angle diversity losses so the ensembled augmented views improve knowledge distillation.
Problem
Knowledge distillation benefits from diverse teacher views, but obtaining diversity normally requires training multiple teacher networks, which is computationally expensive; prior single-teacher augmentation (TeKAP) relies on random noise with little control over diversity.
Approach
Attach N orthogonally initialized linear feature and logit heads (with dropout) to one teacher and optimize them with a constrained inter-angle loss (maximize pairwise angular separation within a learnable margin around the teacher) and an intra-angle loss (spread offset vectors evenly), then distill the (N+1)-way ensemble into the student.
Key results
- On CIFAR-100 (ResNet32x4 teacher, ResNet8x4 student), logit+feature Angular-KD reaches 76.46% vs 75.46% without augmentation and 75.98% with TeKAP.
- On ImageNet (ResNet34 to ResNet18, logit-level), Angular-KD reaches 71.07% top-1 vs 70.41% without augmentation and 70.67% with TeKAP.
- With a single WideRN-40-2 teacher, Angular-KD (76.33%) matches or beats multi-teacher methods such as ensemble distillation (76.31%) and DGKD (76.24%) at lower teacher parameter count.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Angular-KD logit-level augmentation instead of TeKAP | KD on CIFAR-100, RN32×4 teacher and RN8×4 student · CIFAR-100, logit-level KD | top-1 accuracy: 74.79 -> 76.08 | Helped |
| add Angular-KD to KD + CRD instead of TeKAP | KD + CRD on CIFAR-100, RN32×4 teacher and RN8×4 student · CIFAR-100, logit- and feature-level augmentation | top-1 accuracy: 75.98 -> 76.46 | Helped |
| add Angular-KD logit-level augmentation instead of TeKAP | KD on ImageNet, ResNet34 teacher and ResNet18 student · ImageNet validation | Top-1 accuracy: 70.67 -> 71.07 | Helped |
| add constrained inter-angle and intra-angle diversity losses (ablation) | Angular-KD with KD + CRD · CIFAR-100, RN32×4 teacher and RN8×4 student | accuracy: 75.46 -> 76.46 | Helped |
| add intra-angle diversity loss (ablation) | Angular-KD with inter-angle loss · CIFAR-100, RN32×4 teacher and RN8×4 student | accuracy: 76.16 -> 76.46 | Helped |
| add orthogonal initialization to the augmentation heads (ablation) | Angular-KD view augmentation heads with dropout · CIFAR-100, RN32×4 teacher and RN8×4 student | accuracy: 76.35 -> 76.46 | Helped |
| add input dropout to the augmentation heads (ablation) | Angular-KD view augmentation heads with orthogonal initialization · CIFAR-100, RN32×4 teacher and RN8×4 student | accuracy: 76.31 -> 76.46 | Helped |
| use 5 augmentations instead of 1 (ablation) | Angular-KD with KD + CRD · CIFAR-100, RN32×4 teacher and RN8×4 student | accuracy: 75.87 -> 76.46 | Helped |
| set angular margin γ to 0.1 instead of 0.2 (ablation) | Angular-KD constrained inter-angle diversity loss · CIFAR-100, RN32×4 teacher and RN8×4 student | accuracy: 76.46 -> 76.34 | Hurt |
| replace view augmentation with data augmentation (ablation) | KD + CRD · CIFAR-100, RN32×4 teacher and RN8×4 student | accuracy: 75.46 -> 66.91 | Hurt |
Surprising findings
- Ensembling the teacher with TeKAP's random-perturbation views lowers accuracy relative to the teacher alone, whereas Angular-KD's ensemble improves it.
- Replacing view augmentation with standard data augmentation (flip, color jitter, rotation) drops CIFAR-100 student accuracy to 66.91%, far below the 75.46% baseline.
Limitations
- Augmented views are bounded by the single teacher's knowledge and cannot add fundamentally new semantic information.
- Generating multiple views still adds non-negligible training-time overhead.
- Angular perturbations are not semantically grounded, which may limit interpretability and raise adversarial susceptibility (acknowledged in social impact section).
Experimental setup
Datasets: CIFAR-100 ImageNet Imbalanced CIFAR-100 STL-10 TinyImageNet Carvana Image Masking
Benchmarks: CIFAR-100 ImageNet Imbalanced CIFAR-100 STL-10 TinyImageNet Carvana Image Masking
Metrics: top-1 accuracy top-5 accuracy Dice loss IoU
Baselines: TeKAP Logit Distillation (KD) Feature Distillation (CRD) TAKD DGKD Ensemble Distillation ReviewKD DKD MLKD
Builds on: TeKAP Hinton logit distillation CRD DKD MLKD
Closest papers by meaning
- Single Teacher, Multiple Perspectives: Teacher Knowledge Augmentation for Enhanced Knowledge Distillation (ICLR 2025 · similarity 0.93)
- Online Knowledge Distillation with Diverse Peers (AAAI 2020 · similarity 0.89)
- Adaptive Hierarchy-Branch Fusion for Online Knowledge Distillation (AAAI 2023 · similarity 0.89)
- How to Trade Off the Quantity and Capacity of Teacher Ensemble: Learning Categorical Distribution to Stochastically Employ a Teacher for Distillation (AAAI 2024 · similarity 0.89)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.88)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.