Research map / Label-efficient and robust learning / Knowledge distillation
Neural Collapse Inspired Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Neural Collapse-inspired Knowledge Distillation (NCKD), which distills the teacher's class-centroid alignment and simplex-ETF structure into the student and uses normalized prototypes as a classifier, improving accuracy on CIFAR-100, ImageNet and MS-COCO.
Problem
The knowledge gap between teacher and student limits knowledge distillation effectiveness; it is unclear whether existing KD methods induce a good neural collapse (simplex ETF) structure in the student, and whether explicitly transferring it helps.
Approach
Transfer the teacher's neural collapse structure to the student: a contrastive loss aligning student features with teacher class centroids (NC1), a loss matching the student's normalized centroid Gram matrix to the teacher's simplex ETF (NC2), and a classifier built from normalized prototypes (NC3) to cut computation.
Key results
- On CIFAR-100 the method reaches an average top-1 of 75.10% across six teacher-student pairs, the best among the compared methods (SimKD 74.55%, MLKD 74.87%).
- On ImageNet ResNet18 with ResNet34 teacher, top-1 reaches 72.44% versus 71.88% for DIST and 70.66% for KD.
- Plugging the NC losses into CRD raises its CIFAR-100 average from 73.45% to 75.53%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add NC losses (NC1 contrastive prototype alignment, NC2 ETF matching, NC3 normalized-prototype classifier) to the distillation framework | NCKD vs. prior KD methods · CIFAR-100 (6 teacher-student pairs) | top-1 accuracy average: NCKD 75.10 vs SimKD 74.55 and CRD 73.45 | Helped |
| Plug NC losses into CRD as a plug-in module | CRD · CIFAR-100 (6 pairs) | top-1 accuracy average: +2.08 (73.45 to 75.53) | Helped |
| Plug NC losses into SimKD as a plug-in module | SimKD · CIFAR-100 (6 pairs) | top-1 accuracy average: +0.75 (74.55 to 75.30) | Helped |
| Add NC losses to the distillation on ImageNet ResNet18 student from ResNet34 teacher | ResNet18 student · ImageNet top-1 | top-1 accuracy: 72.44 vs DisWOT 72.08 and TTM 72.09 | Helped |
| Add NC losses on ImageNet with MobileNet student from ResNet50 teacher (cross-architecture) | MobileNet student · ImageNet top-1 | top-1 accuracy: 73.61 vs DisWOT 73.22 | Helped |
| Remove NC2 (ETF matching) loss from NCKD (ablation) | NCKD · CIFAR-100 ResNet-8x4 student | top-1 accuracy: 77.23 full vs 75.98 without NC2 on ResNet-8x4 (-1.25) | Helped |
| Remove NC3 classifier (use standard classifier with NC1+NC2) (ablation) | NCKD · CIFAR-100 ResNet-8x4 student | top-1 accuracy: 77.23 full vs 77.00 without NC3 on ResNet-8x4 (+0.23 from NC3) | Helped |
| Add standard KD loss on top of NCKD (ablation) | NCKD · CIFAR-100 ResNet-8x4 student | top-1 accuracy: 77.23 to 77.41 (+0.18) | Helped |
| Replace the NC3 classifier with a standard linear classifier and measure training time (ablation) | NCKD · CIFAR-100 training efficiency | training time per epoch: training time significantly reduced (exact values in right-hand table not extracted) | Helped |
| Use NC3-style reused classifier (SimKD) vs standard KD | SimKD · ImageNet ResNet34/ResNet18 NC3 metric | NC3 metric (closer to 1 is better): 1.11 vs 1.47 for KD | Helped |
Surprising findings
- Removing the NC2 (ETF) loss hurt more than removing the NC3 classifier component, with NC2 identified as the most impactful term.
- The NC3 normalized-prototype classifier matched or outperformed the standard linear classifier while reducing training time per epoch.
Limitations
- Focuses on distillation from a pre-trained teacher; mutual distillation is left to future work.
- Choosing the best teacher for a given student via NC-based criteria is not addressed.
Experimental setup
Datasets: CIFAR-100 ImageNet-1k MS-COCO 2017 STL-10 Tiny-ImageNet
Benchmarks: CIFAR-100 ImageNet-1k MS-COCO STL-10 Tiny-ImageNet
Metrics: top-1 accuracy top-5 accuracy mAP AP50 AP75 NC1 NC2 NC3
Baselines: KD DKD DIST MLKD FitNet RKD PKT CRD ReviewKD NORM SimKD TopKD
Builds on: knowledge distillation CRD SimKD neural collapse
Closest papers by meaning
- Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation (NeurIPS 2022 · similarity 0.92)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.90)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.89)
- PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning (ICML 2026 · similarity 0.89)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.