Research map / Label-efficient and robust learning / Knowledge distillation
VkD: Improving Knowledge Distillation using Orthogonal Projections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VkD, a feature distillation method that uses an orthogonal (Stiefel-manifold) projection layer plus task-specific standardisation or whitening of teacher features, improving ImageNet, COCO detection, and data-limited GAN distillation.
Problem
Feature distillation relies on heuristic projectors and auxiliary losses that add hyperparameters and can distort or shortcut the student representation, hurting generality across tasks, modalities, and architectures.
Approach
Derive from preserving intra-batch feature similarity that the projector should be orthogonal, implement it cheaply via matrix exponential of a skew-symmetric matrix truncated to rows, and replace auxiliary diversity losses with whitening of teacher features in generative tasks.
Key results
- On ImageNet-1K with DeiT-Ti student and RegNetY-160 teacher, VkD-Ti reaches 78.3% top-1, 6.1pp above DeiT-Ti baseline and 3.3pp above USKD (4.4% relative).
- On COCO2017 with ViDT, VkD improves AP by 2.6pp for Swin-nano, 2.1pp for Swin-tiny, and 1pp for Swin-small.
- On CIFAR-100 at 10% data, whitening reduces FID to 16.87 versus 18.03 for KD-DLGAN (and 35.39 for the baseline).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an orthogonal projection instead of the standard distillation setup (ablation) | DeiT-Ti with RegNetY-160 teacher · ImageNet-1K | Top-1 accuracy: 76.3 -> 77.9 | Helped |
| Add task-specific normalisation alongside the orthogonal projection (ablation) | DeiT-Ti with RegNetY-160 teacher · ImageNet-1K | Top-1 accuracy: 77.9 -> 78.3 | Helped |
| Use teacher-feature whitening (ablation) | DA + VkD with BigGAN · CIFAR-10, 10% data | FID: 16.42 ± 0.07 -> 13.16 ± 0.06 | Helped |
| Use teacher-feature whitening (ablation) | DA + VkD with BigGAN · CIFAR-100, 10% data | FID: 24.92 ± 0.15 -> 16.87 ± 0.09 | Helped |
| Use teacher-feature whitening (ablation) | DA + VkD with BigGAN · CIFAR-100, 20% data | FID: 17.97 ± 0.17 -> 14.00 ± 0.07 | Helped |
| Use an orthogonal projection for ViDT object detection | ViDT Swin-nano · COCO2017 val | AP: 40.4 -> 43.0 | Helped |
| Use an orthogonal projection for ViDT object detection | ViDT Swin-tiny · COCO2017 val | AP: 44.8 -> 46.9 | Helped |
| Use an orthogonal projection for ViDT object detection | ViDT Swin-small · COCO2017 val | AP: 47.5 -> 48.5 | Helped |
| Replace token matching with VkD | ViDT Swin-nano student with ViDT-base teacher · COCO2017 val | AP: 41.9 -> 43.0 | Helped |
| Replace token matching with VkD | ViDT Swin-tiny student with ViDT-small teacher · COCO2017 val | AP: 45.8 -> 45.9 | No clear effect |
| Use VkD-Ti rather than SRD | DeiT-Ti with RegNetY-160 teacher · ImageNet-1K | Top-1 accuracy: 77.2 -> 78.3 | Helped |
| Replace VkD with DA + VkD w/ whitening (ablation) | BigGAN generator · CIFAR-100, 100% data | FID: 10.61 ± 0.08 -> 10.41 ± 0.01 | Helped |
Surprising findings
- A linear projection layer eventually outperforms MLP and projector ensembles at long training schedules, as expressive projectors learn representations not shared by the student.
Limitations
- Generative-task gains rely on whitening, which is shown to be necessary; orthogonal projection alone gives weaker results in the generative setting.
- Evaluation is limited to vision tasks (classification, detection, GAN-based generation); no language or other modalities tested.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K MS-COCO 2017 CIFAR-10 CIFAR-100
Benchmarks: ImageNet-1K MS-COCO 2017 val CIFAR-10 CIFAR-100
Metrics: top-1 accuracy AP FPS FID
Baselines: DeiT DeiT with distillation token CivT MaskedKD Manifold DearKD USKD SRD ViDT token matching KD-DLGAN FitNets SPKD
Builds on: DeiT KD-DLGAN ViDT Projector ensemble
Closest papers by meaning
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy (ICLR 2018 · similarity 0.90)
- Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · similarity 0.89)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.89)
- DFD: Distilling the Feature Disparity Differently for Detectors (ICML 2024 · similarity 0.88)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.88)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.