Research map / Label-efficient and robust learning / Knowledge distillation
Multi-label Self Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L2D, a multi-label knowledge distillation method combining one-vs-all multi-label logits distillation with class-aware and instance-aware label-wise embedding distillation, outperforming standard KD methods on VOC, MS-COCO and NUS-WIDE.
Problem
Standard knowledge distillation (logit-based softmax KD and feature-map KD) does not transfer well to multi-label learning, where label probabilities do not sum to one and whole-image feature maps neglect minor classes.
Approach
Decompose multi-label logits into per-label binary distributions for KL distillation, and distill the pairwise structure of label-wise embeddings both within classes across images (CD) and across classes within an image (ID).
Key results
- On MS-COCO with ResNet-101 teacher and ResNet-34 student, L2D reaches 72.87 mAP versus 70.31 for the vanilla student and 70.86 for the best baseline PS.
- On VOC 2007 with ResNet-50 teacher and ResNet-18 student, L2D achieves 85.71 mAP versus 84.01 vanilla and 84.48 for MLD.
- Ablation on MS-COCO: adding CD gives +1.6 mAP and ID gives +1.48 mAP over the MLD-only variant; full model reaches 72.87 mAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply L2D instead of MLD | ResNet-18 student distilled from ResNet-50 · Pascal VOC 2007 | mAP: 84.48 -> 85.71 | Helped |
| apply L2D instead of MLD | ResNet-34 student distilled from ResNet-101 · MS-COCO | mAP: 70.68 -> 72.87 | Helped |
| apply L2D instead of MLD | ResNet-34 student distilled from ResNet-101 · NUS-WIDE validation | mAP: 54.44 -> 55.31 | Helped |
| add multi-label logits distillation (MLD) (ablation) | ResNet-34 student · MS-COCO | mAP: 70.31 -> 70.68 | Helped |
| add class-aware label-wise embedding distillation (CD) to MLD (ablation) | ResNet-34 student · MS-COCO | mAP: 70.68 -> 71.91 | Helped |
| add instance-aware label-wise embedding distillation (ID) to MLD (ablation) | ResNet-34 student · MS-COCO | mAP: 70.68 -> 71.79 | Helped |
| combine MLD, CD, and ID (ablation) | ResNet-34 student · MS-COCO | mAP: 71.91 -> 72.87 | Helped |
Surprising findings
- With WRN-101 as teacher, the L2D student (76.61 mAP) slightly exceeds the teacher (74.70 mAP) on MS-COCO, which the authors attribute to alleviating teacher overfitting.
- Conventional feature-based KD methods (RKD, PKT, ReviewKD, MSE) yield only minor gains or none over the vanilla student in multi-label settings.
Limitations
- Gains on VOC are smaller because VOC has fewer labels per image, so conventional KD methods perform relatively better there.
- The label-wise embedding encoder with per-class queries and cross-attention adds architectural requirements compared with plain backbones.
Experimental setup
Datasets: Pascal VOC 2007 MS-COCO 2014 NUS-WIDE ImageNet
Benchmarks: Pascal VOC 2007 MS-COCO 2014 NUS-WIDE
Metrics: mAP OF1 CF1
Baselines: RKD PKT ReviewKD MSE PS MLD
Builds on: knowledge distillation (Hinton et al.) ML-Decoder Query2Label ReviewKD RKD
Closest papers by meaning
- Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels (ICLR 2025 · similarity 0.86)
- Distilling Reliable Knowledge for Instance-Dependent Partial Label Learning (AAAI 2024 · similarity 0.86)
- Rebalancing Multi-Label Class-Incremental Learning (AAAI 2025 · similarity 0.85)
- FreeKD: Knowledge Distillation via Semantic Frequency Prompt (CVPR 2024 · similarity 0.85)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.85)
- Scaled Decoupled Distillation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.