Research map / Label-efficient and robust learning / Knowledge distillation
PLD: A Choice-Theoretic List-Wise Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Recasts logit-based knowledge distillation as a Plackett-Luce list-wise ranking loss that places the true label first and orders the remaining classes by teacher logits, weighting each step by teacher softmax mass.
Problem
Logit-based knowledge distillation adds a distillation term (KL or correlation-based) to cross-entropy, which requires manually tuned mixing weights; the paper seeks a unified objective whose weights come from the teacher's confidence.
Approach
Fix a teacher-optimal permutation (true label first, rest sorted by descending teacher logits) and minimize its Plackett-Luce negative log-likelihood on student logits, with each selection step weighted by the teacher's softmax probability of that class, yielding a convex, translation-invariant loss that subsumes weighted cross-entropy, ListMLE and P-ListMLE.
Key results
- On ImageNet-1K ResNet-50 student from ResNet-152 teacher, PLD reaches 77.30% Top-1 vs 76.60% DIST and 76.80% KD (best tuned).
- On CIFAR-100 homogeneous WRN-40-2 to WRN-40-1, PLD reaches 73.50% vs 72.74% for DIST and 72.44% for KD; heterogeneous ResNet-50 to MobileNetV2 gives 68.16% vs DKD 69.51% (DKD better).
- On MS-COCO, PLD matches or slightly beats DKD on several Faster R-CNN pairs (e.g., ResNet-101 to ResNet-18-FPN: 32.47 AP vs 32.11 AP) but is lower on ResNet-50 to MobileNetV2-FPN (28.91 vs 29.24 AP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the KD or DIST objective with PLD | ImageNet-1K homogeneous teacher–student pairs · ImageNet-1K, Top-1 accuracy | Top-1 accuracy: +0.42 pp versus DIST and +1.04 pp versus KD | Helped |
| Replace KD or DIST with PLD | ResNet-50 student distilled from diverse teachers · ImageNet-1K, Top-1 accuracy | Top-1 accuracy: +0.48 pp versus DIST; +1.09 pp versus KD | Helped |
| Replace DKD with PLD | Faster R-CNN detectors · MS-COCO val2017, AP across three teacher–student pairs | AP: 29.24 -> 28.91; 32.11 -> 32.47; 36.54 -> 36.60 | Mixed |
| Add CE mixing weight 0.10 rather than no CE (ablation) | KD and DIST · ImageNet-1K, ResNet-152 → ResNet-50 | Top-1 accuracy: KD: 75.92 -> 76.80; DIST: 76.47 -> 76.60 | Helped |
| Set PLD teacher-softmax temperature to 0.50 instead of 1.00 (ablation) | PLD · ImageNet-1K, ResNet-152 → ResNet-50 | Top-1 accuracy: 77.30 -> 76.06 | Hurt |
| Set PLD teacher-softmax temperature to 1.50 instead of 1.00 (ablation) | PLD · ImageNet-1K, ResNet-152 → ResNet-50 | Top-1 accuracy: 77.30 -> 77.17 | Hurt |
| Set PLD teacher-softmax temperature to 2.00 instead of 1.00 (ablation) | PLD · ImageNet-1K, ResNet-152 → ResNet-50 | Top-1 accuracy: 77.30 -> 76.46 | Hurt |
| Set PLD teacher-softmax temperature to 4.00 instead of 1.00 (ablation) | PLD · ImageNet-1K, ResNet-152 → ResNet-50 | Top-1 accuracy: 77.30 -> 75.22 | Hurt |
| Replace teacher-confidence weights with uniform ListMLE weights (ablation) | PLD · ImageNet-1K, ResNet-152 → ResNet-50 | Top-1 accuracy: 77.30 -> 74.11 | Hurt |
| Use position-aware p-ListMLE weights instead of PLD confidence weights (ablation) | PLD · ImageNet-1K, ResNet-152 → ResNet-50 | Top-1 accuracy: 77.30 -> 76.60 | Hurt |
| Standardize teacher and student logits before PLD (ablation) | PLD · ImageNet-1K, MobileNetV4-Small student | Top-1 accuracy: 71.16 -> 70.66 | Hurt |
| Standardize only teacher logits for PLD weights (ablation) | PLD · ImageNet-1K, MobileNetV4-Small student | Top-1 accuracy: 71.16 -> 70.81 | Hurt |
Surprising findings
- On ImageNet-1K distribution matching, PLD yields higher KL divergence to the teacher than DIST on several homogeneous and heterogeneous pairs (e.g., ViT-Large to ViT-Base 0.72 vs 0.47), despite better accuracy.
- Standardizing logits degrades PLD (71.16% to 70.66%) while it modestly helps DIST.
Limitations
- Requires identical class vocabularies between teacher and student; does not support mismatched label sets or incremental class addition.
- Sorting adds O(C log C) cost per example versus O(C) for KD/DIST.
- Benefit may diminish when teacher softmax is nearly uniform.
Experimental setup
Datasets: CIFAR-100 ImageNet-1K MS-COCO
Benchmarks: CIFAR-100 ImageNet-1K MS-COCO
Metrics: Top-1 accuracy Top-5 accuracy AP AP50 AP75
Baselines: KD DIST DKD AT FitNet PKT RKD SP VID ListMLE P-ListMLE
Builds on: knowledge distillation ListMLE P-ListMLE DIST DKD
Closest papers by meaning
- Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.90)
- Maintaining Fairness in Logit-based Knowledge Distillation for Class-Incremental Learning (AAAI 2025 · similarity 0.90)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.90)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.89)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.89)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.