Research map / Label-efficient and robust learning / Knowledge distillation
All You Need in Knowledge Distillation Is a Tailored Coordinate System
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TCS, a teacher-free distillation method that projects student features into a PCA coordinate system of a (self-supervised) teacher's features and selects a subset of dimensions via iterative feature selection, plus an efficient LSH-based loss (eLSH).
Problem
Knowledge distillation normally requires a large teacher trained on the target task, which is inflexible and costly (nearly doubling training time and GPU memory); SSL-pretrained teachers are rarely used effectively.
Approach
The dark knowledge of a teacher is captured by the coordinate system (PCA subspace) of its features computed in one forward pass on target data; the student's features are aligned to this coordinate system and a learned mask tailors it by selecting dimensions, with all post-backbone operations linear so no inference overhead.
Key results
- On CIFAR-100 with ResNet18 student and ViT-S teacher, TCS reaches 81.37% top-1 versus 80.50% for self-distillation and 77.26% for KD, with runtime about 32 s/epoch versus 385 s for self-distillation.
- On ImageNet-1K with ResNet18 student, TCS achieves 71.99% with Swin-T teacher, versus 71.14% for KD, with roughly 5.8 GB memory versus 7.5 GB for KD.
- In 16-shot pFSL on CUB with DINOv2 ViT-S, TCS with ViT-L teacher gives 87.5% top-1 accuracy versus 87.0% baseline without teacher distillation (the paper's ablation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TCS instead of the strongest listed KD baseline | ResNet18 student with ViT-S teacher · CIFAR-100 | accuracy: 80.50 -> 81.37 | Helped |
| Use TCS instead of OFA | ResNet18 student with ConvNeXt-T teacher · CIFAR-100 | accuracy: 80.91 -> 81.60 | Helped |
| Use TCS instead of OFA | ResNet18 student with Mixer-B/16 teacher · CIFAR-100 | accuracy: 79.39 -> 81.02 | Helped |
| Use TCS instead of OFA | ResNet18 student with DeiT-T teacher · ImageNet-1K val | accuracy: 71.34 -> 71.40 | Helped |
| Use TCS instead of OFA | ResNet18 student with Swin-T teacher · ImageNet-1K val | accuracy: 71.85 -> 71.99 | Helped |
| Use TCS instead of OFA | ResNet18 student with Mixer-B teacher · ImageNet-1K val | accuracy: 71.38 -> 71.45 | Helped |
| Use TCS instead of IbM2 | ViT-S student with DINOv2 ViT-B teacher · CUB 16-shot | top-1 accuracy: 86.5 ± 0.3 -> 87.6 ± 0.0 | Helped |
| Use TCS instead of IbM2 | ViT-S student with MoCov3 ViT-B teacher · ImageNet-1K 16-shot | top-1 accuracy: 60.3 ± 0.2 -> 61.6 ± 0.1 | Helped |
| Add a ViT-S teacher to the ViT-B student (ablation) | TCS with MoCov3 · CUB 16-shot | top-1 accuracy: $67.32 \pm 0.21$ -> $71.17 \pm 0.26$ | Helped |
| Compute PCA using 3-shot in-domain data instead of no PCA (ablation) | TCS on CUB · CUB 16-shot | top-1 accuracy: $65.30 \pm 0.30$ -> $70.37 \pm 0.11$ | Helped |
| Compute PCA using 16-shot in-domain data instead of 3-shot in-domain data (ablation) | TCS on CUB · CUB 16-shot | top-1 accuracy: $70.37 \pm 0.11$ -> $71.80 \pm 0.05$ | Helped |
| Replace PCA coordinates with random vectors (ablation) | TCS on CUB · CUB 16-shot | top-1 accuracy: $65.30 \pm 0.30$ -> $65.71 \pm 0.61$ | Hurt |
Surprising findings
- Out-of-domain PCA coordinate systems (e.g., computed from Diabetic Retinopathy data) still improve accuracy by roughly 2% over the baseline without TCS.
- A small teacher (ViT-S) can help a larger student (ViT-B) by 2-4% accuracy, suggesting large capacity gaps are not harmful for TCS.
Limitations
- Evaluated only on image classification; object detection and segmentation are left for future work.
- PCA is unreliable when the number of shots is very small (1-2 shots), where TCS underperforms IbM2 in some cases.
- Theoretical foundation of how dark knowledge is encoded in the coordinate system is not established.
Experimental setup
Datasets: CIFAR-100 ImageNet-1K CUB-200-2011 VTAB-1K (Flowers102, Pets, Diabetic Retinopathy)
Benchmarks: CIFAR-100 ImageNet-1K CUB-200-2011
Metrics: top-1 accuracy GPU memory (GB) runtime (s/epoch)
Baselines: KD DKD CRD DIST OFA LSHL2 Self-distillation IbM2
Builds on: CRD LSH feature mimicking (LSHL2) IbM2 (practical few-shot learning) self-distillation
Closest papers by meaning
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.91)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.90)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.89)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.89)
- Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher (ICLR 2025 · similarity 0.89)
- Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.