Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation with Auxiliary Variable
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AuxKD, a knowledge distillation method that introduces an instance-membership auxiliary variable to reformulate the student's predictive distribution, unifying logit and feature matching into one objective.
Problem
Conventional knowledge distillation makes the student mimic the teacher's predictive distribution with the same modeling strategy, which causes sub-optimal transfer due to the teacher-student capacity gap; existing fixes need complicated or student-specific teacher training.
Approach
Introduce instance membership as a label-related latent auxiliary variable to reformulate the student's predictive distribution, yielding a single contrastive-plus-vMF objective that unifies logit-level and feature-level distillation, and removes the parametric softmax classifier at inference.
Key results
- On ImageNet with ResNet34 teacher and ResNet18 student, AuxKD reaches 73.04% Top-1, surpassing DiffKD (72.22%) by 0.82%.
- On CIFAR-100 self-distillation with PS-KD plug-in, AuxKD improves Top-1 by 0.86%-1.24% over IPWD across four architectures (e.g., DenseNet121 84.24 on ResNeXt29 vs 83.30 IPWD).
- Linear probing with WRN-40-2 teacher and WRN-16-2 student: AuxKD gets 74.4% on STL-10 and 39.0% on Tiny-ImageNet, outperforming DKD by 1.5% and 1.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AuxKD instead of DiffKD | ResNet18 distilled from ResNet34 · ImageNet-1K val | Top-1 accuracy: 72.22 -> 73.04 | Helped |
| Use AuxKD instead of DiffKD | ResNet18 distilled from ResNet34 · ImageNet-1K val | Top-5 accuracy: 90.64 -> 90.88 | Helped |
| Use AuxKD instead of DiffKD | MobileNetV1 distilled from ResNet50 · ImageNet-1K val | Top-1 accuracy: 73.62 -> 73.90 | Helped |
| Use AuxKD instead of DiffKD | MobileNetV1 distilled from ResNet50 · ImageNet-1K val | Top-5 accuracy: 91.42 -> 91.51 | Helped |
| Use AuxKD instead of IPWD | DenseNet121 self-distillation · CIFAR-100 | Top-1 accuracy: 81.60 -> 82.84 | Helped |
| Use AuxKD instead of SSKD | WRN-16-2 distilled from WRN-40-2 · STL-10 linear probing | Top-1 accuracy: 73.6 -> 74.4 | Helped |
| Use AuxKD instead of CoCoRD | WRN-16-2 distilled from WRN-40-2 · Tiny-ImageNet linear probing | Top-1 accuracy: 38.4 -> 39.0 | Helped |
Limitations
- Explores only one realization of the auxiliary variable (instance membership) and one parameterization scheme.
- Pipeline requires maintaining a teacher-feature queue (MoCo-style), adding implementation complexity.
Experimental setup
Datasets: CIFAR-100 ImageNet-1K STL-10 Tiny-ImageNet PASCAL-VOC MS-COCO
Benchmarks: CIFAR-100 ImageNet-1K STL-10 Tiny-ImageNet PASCAL-VOC MS-COCO
Metrics: Top-1 accuracy Top-5 accuracy AP AP50 AP75
Baselines: KD DKD IPWD WSLD MLD DIST ReviewKD CRD WCoRD CoCoRD NORM DiffKD
Builds on: Hinton knowledge distillation (logit matching) MoCo Shen et al. 2021 contrastive clustering PS-KD MINE
Closest papers by meaning
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.91)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.89)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.88)
- AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation (ICLR 2024 · similarity 0.88)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.88)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.