Research map / Label-efficient and robust learning / Knowledge distillation
Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a plug-and-play, tanh-relaxed Kendall's tau ranking loss on teacher-student logit channel orderings to existing logit-based KD methods, improving CIFAR-100, ImageNet, COCO and ViT distillation results.
Problem
Logit-based knowledge distillation with KL divergence gives small gradients to low-probability channels and can converge to suboptimal solutions where the student misclassifies despite low KL loss.
Approach
Adds an auxiliary differentiable (tanh-relaxed) Kendall's tau ranking loss that constrains the pairwise ordering of student logits to match the teacher's; the loss is largely invariant to logit scale, so it complements the KL term rather than replacing it.
Key results
- Adding the ranking loss to vanilla KD improves CIFAR-100 accuracy by 1.83% on average, e.g., ResNet32x4 to ResNet8x4 from 73.33 to 74.74 (+1.41).
- On ImageNet (ResNet50 to MobileNet-V1), KD+Ours reaches 71.54 top-1 vs 70.50 for KD (+1.04).
- On COCO object detection (ResNet50 to MobileNet-V2), KD+Ours reaches 31.99 AP vs 30.13 for KD (+1.86).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Kendall’s τ ranking loss to logit-based distillation | KD and other logit-based methods · CIFAR-100 heterogeneous and homogeneous teacher-student pairs | Top-1 accuracy (%): +0.28 to +3.10 in heterogeneous pairs; +0.09 to +2.2 in homogeneous pairs | Helped |
| Add Kendall’s τ ranking loss to KD | ResNet50→MN-V1 · ImageNet validation | Top-1 and Top-5 accuracy (%): +1.04 on Top-1; +1.04 on Top-5 | Helped |
| Add Kendall’s τ ranking loss to KD | ResNet56 teacher with ViT students · CIFAR-100 validation | Top-1 accuracy (%): +2.14, +0.77, +0.30, +0.96 | Helped |
| Add Kendall’s τ ranking loss to KD | ResNet50→MobileNet-V2 · COCO2017 | AP, AP50, AP75: +1.86 AP; +3.52 AP50; +2.02 AP75 | Helped |
| Add ranking loss to DIST | DIST · CIFAR-100 teacher-student pairs | Top-1 accuracy (%): +0.27, +0.52, +0.23, +0.26, +0.13 | Helped |
| Add ranking loss to CTKD and NKD | CTKD and NKD · ImageNet validation, ResNet34→ResNet18 | Top-1 and Top-5 accuracy (%): CTKD: +0.36 Top-1, +0.03 Top-5; NKD: +0.10 Top-1, +0.22 Top-5 | Helped |
| Vary ranking-loss weight γ (ablation) | KD · CIFAR-100, four teacher-student configurations | Top-1 accuracy (%): γ=0.1: 74.15, 74.15, 75.52, 69.25; γ=0.5: 74.84, 74.07, 76.34, 69.81; γ=0.9: 74.74, 74.49, 76.58, 70.45; γ=2: 74.62, 74. | Mixed |
| Vary Kendall approximation steepness k (ablation) | KD · CIFAR-100, four teacher-student configurations | Top-1 accuracy (%): k=0.1: 73.13, 73.75, 75.47, 68.9; k=0.5: 74.36, 74.17, 75.73, 68.87; k=1: 74.74, 74.49, 76.58, 70.45; k=2: 74.79, 74.87, | Mixed |
| Remove normalization from ranking loss (ablation) | KD+Ours · CIFAR-100, two teacher-student pairs | Top-1 accuracy (%): 76.38(+2.31); 74.07(+0.53), versus normalized 75.98(+1.91); 74.49(+0.95) | Mixed |
| Use Form1, Form2, or Form3 ranking loss instead of the main form (ablation) | KD · CIFAR-100, similar and different structures | Average Top-1 accuracy (%): Similar: +1.46, +1.41, +1.37; different: +0.76, +0.62, +0.31 | Helped |
| Add ranking loss at different KD temperatures (ablation) | KD · CIFAR-100 validation, ResNet32×4→ResNet8×4 | Top-1 accuracy (%): T=4: +0.23; T=5: +0.10; T=6: +0.93; T=10: +0.49 | Helped |
| Use ranking loss alone rather than KL alone (ablation) | KD · CIFAR-100, five teacher-student pairs | Top-1 accuracy (%): 75.60 vs 74.07; 78.04 vs 77.70; 75.81 vs 74.83; 73.72 vs 72.98; 74.00 vs 73.54 | Helped |
Surprising findings
- Using the ranking loss alone achieves performance comparable to KL divergence alone on CIFAR-100 (e.g., 75.60 vs 74.07 on ResNet32x4 to ShuffleNet-V1 with only ranking loss vs KD-only).
- Min-k channel ranking loss sometimes gives better gains than top-k channel ranking loss on some teacher-student pairs.
Limitations
- Gains on some strong baselines are small (e.g., DKD+Ours +0.10 on one CIFAR-100 pair).
- Ranking loss uses a tanh steepness parameter k and loss weight gamma that must be tuned; performance varies notably across settings.
- Theoretical claims about gradient scale and optimal-solution domain are argued informally and the ranking loss uses normalization that the authors add for stability.
Experimental setup
Datasets: CIFAR-100 ImageNet MS-COCO
Benchmarks: CIFAR-100 ImageNet MS-COCO
Metrics: top-1 accuracy top-5 accuracy AP AP50 AP75
Baselines: KD FitNet CRD ReviewKD DIST LSKD CTKD DKD MLKD AT OFD FGFI
Builds on: Knowledge distillation (Hinton) DIST CTKD DKD MLKD
Closest papers by meaning
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.90)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.89)
- Maintaining Fairness in Logit-based Knowledge Distillation for Class-Incremental Learning (AAAI 2025 · similarity 0.89)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.89)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.88)
- Scaled Decoupled Distillation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.