Research map / Label-efficient and robust learning / Knowledge distillation
DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeepKD, a logit-based distillation trainer that gives independent GSNR-scaled momentum buffers to task, target-class, and non-target-class gradients and adds a curriculum-style dynamic top-k mask that filters low-confidence non-target logits.
Problem
Existing decoupled distillation methods (DKD, DOT) do not handle the conflict between target-class and non-target-class knowledge flows in optimization, lack theory for momentum allocation, and treat low-confidence non-target logits uniformly, injecting noise.
Approach
Decompose the student gradient into task-oriented, target-class, and non-target-class components, each with its own momentum buffer whose coefficient is tied to its gradient signal-to-noise ratio, plus a dynamic top-k mask over non-target logits whose k grows over training (curriculum learning).
Key results
- On CIFAR-100 homogeneous pairs, MLKD+Ours (w. top-k) reaches 79.15% top-1 for ResNet32x4 teacher / ResNet8x4 student setting family, a gain of +2.07 over MLKD.
- On ImageNet-1K ResNet50->MobileNet-V1, KD+Ours (w. top-k) reaches 74.65% top-1 vs 70.50% for KD.
- On MS-COCO with ResNet50 teacher and MobileNet-V2 student, DKD+Ours† reaches 34.20 AP vs 32.34 for DKD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DeepKD decoupled gradients and dynamic top-k masking to CRLD | CRLD, CIFAR-100 homogeneous (ResNet32×4→ResNet8×4) · CIFAR-100 validation | top-1 accuracy: 77.60 -> 79.25 | Helped |
| Add DeepKD with dynamic top-k masking to CRLD | CRLD, CIFAR-100 heterogeneous (WRN-40-2→MN-V2) · CIFAR-100 validation | top-1 accuracy: +2.48% | Helped |
| Add DeepKD with dynamic top-k masking to CRLD | CRLD, ImageNet-1K (ResNet34→ResNet18) · ImageNet-1K validation | top-1 accuracy: +1.86% | Helped |
| Add DeepKD with dynamic top-k masking to KD | KD, ImageNet-1K (ResNet50→MN-V1) · ImageNet-1K validation | top-1 accuracy: 70.50% vs. 74.65% | Helped |
| Add DeepKD with dynamic top-k masking to DKD | DKD, MS-COCO val2017 · MS-COCO val2017 | AP: +1.86% | Helped |
| Use 0.075 momentum offsets for TOG, TCG, and NCG instead of 0.00 offsets (ablation) | DeepKD, ResNet32×4→ResNet8×4 · CIFAR-100 validation | top-1 accuracy: 74.13 -> 76.69 | Helped |
| Add TASK, TCKD, and NCKD together rather than using NCKD alone (ablation) | DeepKD loss, ResNet32×4→ResNet8×4 · CIFAR-100 validation | top-1 accuracy: 75.25 -> 76.69 | Helped |
| Add NCKD to TASK and TCKD (ablation) | DeepKD loss, ResNet32×4→ResNet8×4 · CIFAR-100 validation | top-1 accuracy: 71.50 -> 76.69 | Helped |
| Add dynamic top-k masking to decoupled KD gradients (ablation) | KD, ResNet32×4→ResNet8×4 · CIFAR-100 validation | top-1 accuracy: 76.69 -> 77.03 | Helped |
| Add decoupled gradients to KD (ablation) | KD, ResNet32×4→ResNet8×4 · CIFAR-100 validation | top-1 accuracy: 73.33 -> 76.69 | Helped |
| Add DTM to KD without decoupled gradients (ablation) | KD, ResNet32×4→ResNet8×4 · CIFAR-100 validation | top-1 accuracy: 73.33 -> 74.89 | Helped |
Surprising findings
- Applying DOT on top of several logit methods (e.g., MLKD+DOT, CRLD+DOT) often lowers accuracy relative to the base method in the reported tables.
Limitations
- Focuses only on logit-based distillation; extension to feature distillation is left as future work.
- Optimal static k value requires ablation or a 20%-data pilot run.
- Gains on some heterogeneous pairs are small (e.g., +0.37 to +0.67 for DKD variants).
Experimental setup
Datasets: CIFAR-100 ImageNet-1K MS-COCO
Benchmarks: CIFAR-100 ImageNet-1K MS-COCO
Metrics: top-1 accuracy top-5 accuracy AP AP50 AP75
Baselines: KD DKD DOT LSKD MLKD CRLD FitNet SimKD CAT-KD ReviewKD
Builds on: DKD DOT Hinton knowledge distillation SGD with momentum CTKD
Closest papers by meaning
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.90)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.89)
- Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.88)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.88)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.88)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.