Research map / Label-efficient and robust learning / Knowledge distillation

Adaptive Dual Guidance Knowledge Distillation

AAAI 2025 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADG-KD, which trains an initialized teacher under bidirectional supervision to form a teacher optimization route and adaptively fuses it with the pre-trained teacher's guidance to distill knowledge into lightweight students.

Problem

Knowledge distillation suffers from the capacity gap between large teachers and small students; existing adaptive KD methods degrade the pre-trained teacher and provide knowledge not dynamically matched to student representation ability.

Approach

Introduce an initialized teacher with the same structure as the pre-trained teacher, optimized by a conditional triplet loss that pulls it toward the student early and toward the pre-trained teacher later (forming a bidirectional optimization route, BOR), and fuse guidance from the pre-trained teacher and BOR per instance with learned importance factors computed from latent teacher representations

Key results

• On CIFAR-100, ADG-KD reaches 77.44% for ResNet32x4 to ResNet8x4 and 78.72% for ResNet32x4 to ShuffleNetV2, outperforming vanilla KD by 1.80-4.11 points on homogeneous pairs and 2.62-4.07 on heterogeneous pairs.
• On ImageNet, ADG-KD gains 1.56% Top-1 over vanilla KD with ResNet34 to ResNet18.
• On MS-COCO with ResNet101 to ResNet50 (Faster R-CNN FPN), ADG-KD improves AP by 2.10 over vanilla KD (36.34 vs 33.97 is for the baseline KD gap; reported AP 36.34).

Measured effects reported in the paper

Surprising findings

• In some teacher-student pairs the student trained with ADG-KD outperforms its pre-trained teacher (negative gap in Table 5).
• The BOR-only variant ADG-KD-NT still outperforms KD, DML and TAKD, indicating the initialized teacher's route alone helps bridging the gap.

Limitations

• Gains on some heterogeneous pairs are smaller than on others, and results on ImageNet are reported only for a few pairs.
• Method adds an extra initialized teacher and loss terms, increasing training cost; compute cost is not analyzed.
• Evaluation is limited to image classification and object detection; applicability to other tasks is untested.

Experimental setup

Datasets: CIFAR-100 ImageNet MS-COCO

Benchmarks: CIFAR-100 ImageNet MS-COCO

Metrics: Top-1 accuracy Top-5 accuracy AP AP50 AP75

Baselines: KD DKD MKD CAT-KD KR CTKD LSKD FitNet CRD WCoRD TAKD DGKD

Builds on: Knowledge distillation (Hinton et al. 2015) DKD TAKD latent factor models

Closest papers by meaning

• All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.91)
• Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
• PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning (ICML 2026 · similarity 0.89)
• Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition (AAAI 2025 · similarity 0.88)
• Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.88)
• Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.88)