Research map / Label-efficient and robust learning / Knowledge distillation
Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTKD-RL, which uses a policy-gradient RL agent to generate per-sample teacher weights for multi-teacher knowledge distillation, using teacher performance and teacher-student gaps as state and student loss as reward.
Problem
Balancing distillation strengths across multiple teachers in multi-teacher knowledge distillation; existing weighting methods use only one aspect (teacher performance or teacher-student gap) and ignore the student's feedback.
Approach
Formulates multi-teacher weight assignment as an RL problem: an agent takes a state combining teacher features, logits, CE loss, and teacher-student feature similarity and KL divergence, outputs per-sample teacher weights, and is updated with policy gradient using a reward based on the student's negative losses.
Key results
- On CIFAR-100 with ShuffleNetV2 student, MTKD-RL reaches 78.39% vs 78.09% for CA-MKD and 72.84% baseline.
- On ImageNet, MTKD-RL improves ResNet-18 to 72.82% (baseline 70.35%) and ResNet-34 to 76.77% (baseline 73.64%).
- Object detection on COCO with ResNet-34 backbone gains 1.5 mAP on average over baseline backbones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MTKD-RL instead of the strongest reported multi-teacher KD baseline | CIFAR-100 student networks · CIFAR-100 | Acc@1: 80.28 -> 80.58; 74.35 -> 74.63; 78.09 -> 78.39; 75.26 -> 75.35 | Helped |
| Use MTKD-RL instead of the strongest reported multi-teacher KD baseline | ResNet-18 and ResNet-34 students · ImageNet | Acc@1: 72.38 -> 72.82; 76.06 -> 76.77 | Helped |
| Use MTKD-RL instead of the strongest reported multi-teacher KD baseline | DeiT-Tiny and CaiT-XXS24 students · ImageNet | Acc@1: 74.35 -> 75.14; 78.65 -> 79.22 | Helped |
| Pretrain the ResNet backbone with MTKD-RL rather than the baseline | ResNet-18 and ResNet-34 detection backbones · COCO-2017 | mAP: 34.1 -> 35.1; 36.5 -> 37.7; 31.6 -> 32.7; 33.5 -> 34.7; 37.6 -> 39.0; 39.6 -> 40.8; 35.2 -> 36.9; 37.0 -> 38.5 | Helped |
| Pretrain the ResNet-34 backbone with MTKD-RL rather than the baseline | DeepLabV3 and PSPNet · Cityscapes, ADE20K, COCO-Stuff | score (metric not specified): 76.34 -> 77.42; 36.08 -> 37.07; 29.97 -> 31.82; 74.60 -> 75.89; 36.84 -> 37.78; 31.25 -> 32.39 | Helped |
| Use teacher performance features, logits, and cross-entropy as the state input (ablation) | MTKD-RL with ShuffleNetV2 · CIFAR-100 | Acc: 1.01% gain over AVER; 77.84 | Helped |
| Use teacher-student feature similarity and logit KL-divergence as the state input (ablation) | MTKD-RL with ShuffleNetV2 · CIFAR-100 | Acc: 0.63% gain; 77.46 | Helped |
| Combine teacher performance and teacher-student gaps in the state (ablation) | MTKD-RL with ShuffleNetV2 · CIFAR-100 | Acc: 1.56% improvement; 78.39 | Helped |
| Replace Policy Gradient with DPG (ablation) | MTKD-RL with ShuffleNetV2 · CIFAR-100 | Acc: 78.39$_{\pm 0.14}$ -> 78.16$_{\pm 0.11}$ | No clear effect |
| Replace Policy Gradient with DDPG (ablation) | MTKD-RL with ShuffleNetV2 · CIFAR-100 | Acc: 78.39$_{\pm 0.14}$ -> 78.05$_{\pm 0.22}$ | No clear effect |
| Replace Policy Gradient with PPO (ablation) | MTKD-RL with ShuffleNetV2 · CIFAR-100 | Acc: 78.39$_{\pm 0.14}$ -> 78.28$_{\pm 0.08}$ | No clear effect |
| Add logit-level KD to the baseline (ablation) | ShuffleNetV2 · CIFAR-100 | Acc@1: 4.14% accuracy improvements | Helped |
Surprising findings
- The multi-teacher weight decision is reported to be insensitive to the choice of RL algorithm (PG, DPG, DDPG, PPO gave similar accuracy).
Limitations
- Gains over the strongest baselines (CA-MKD, MMKD) are small, often under 0.5%, with overlapping standard deviations.
- The paper itself notes the RL algorithm choice matters little, so the RL framing's benefit is unclear.
- Extra memory and time cost for storing episode history and training agents.
Experimental setup
Datasets: CIFAR-100 ImageNet COCO-2017 Cityscapes ADE20K COCO-Stuff-164K
Benchmarks: CIFAR-100 ImageNet COCO-2017 Cityscapes ADE20K COCO-Stuff-164K
Metrics: Top-1 accuracy mAP mIoU
Baselines: AVER (KD) AVER (KD+FitNet) AMTML-KD AEKD CA-MKD MMKD
Builds on: Knowledge distillation (Hinton) FitNet CA-MKD policy gradient AVER multi-teacher KD
Closest papers by meaning
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.88)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.88)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.88)
- Masking Teacher and Reinforcing Student for Distilling Vision-Language Models (CVPR 2026 · similarity 0.87)
- Scaling Multi-Teacher Distillation for Digital Pathology (NeurIPS 2026 · similarity 0.87)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.