Research map / Label-efficient and robust learning / Knowledge distillation
Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MCMI, which fine-tunes a pre-trained teacher to jointly maximize log-likelihood and conditional mutual information so it gives a better Bayes conditional probability estimate for knowledge distillation, improving student accuracy across KD methods, especially in few-shot and zero-shot settings.
Problem
Teachers in knowledge distillation are trained by maximum log-likelihood, which gives a poor estimate of the unknown Bayes conditional probability distribution (BCPD) that the student should learn from, losing contextual information in images.
Approach
Train the teacher to maximize the sum of the empirical log-likelihood of ground-truth labels and the empirical conditional mutual information I(X;Ŷ|Y) (computed with per-class centroids held fixed during fine-tuning), so the teacher captures contextual information while remaining accurate; the resulting teacher is then used unchanged in existing KD frameworks.
Key results
- Replacing the MLL teacher with an MCMI teacher raises student accuracy across 11 KD methods on CIFAR-100, with gains up to 3.32% (AT, ResNet32x4 to ShuffleNetV1).
- In few-shot KD with 5% of training data, MCMI teacher improves KD student accuracy by 5.72% (52.30 to 58.02).
- In zero-shot KD on CIFAR-10 with five omitted classes, the dropped classes' student accuracy rises from 0% to as much as 84% with the MCMI teacher.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the MLL teacher with an MCMI teacher | AT, ResNet-32×4→ShuffleNetV1 · CIFAR-100 | student accuracy: +3.32 | Helped |
| Replace the MLL teacher with an MCMI teacher | KD, ResNet34→ResNet18 · ImageNet validation | Top-1 accuracy: 0.51% | Helped |
| Replace the MLL teacher with an MCMI teacher | KD, ResNet50→MobileNetV2 · ImageNet validation | Top-1 accuracy: 0.60% | Helped |
| Replace the MLL teacher with an MCMI teacher | KD, ResNet-56→ResNet-20 · CIFAR-100 few-shot, 5% | student accuracy: +5.72 | Helped |
| Replace the MLL teacher with an MCMI teacher | CRD, ResNet-56→ResNet-20 · CIFAR-100 few-shot, 5% | student accuracy: +3.80 | Helped |
| Use MCMI teacher instead of MLL teacher | KD student · Synthetic Gaussian, δμ=0.5 | student accuracy: 56.32±0.27 -> 57.02±0.18 | Helped |
| Train teacher with MCMI instead of early stopping | KD, ResNet34→ResNet18 · ImageNet | student accuracy: 71.25 -> 71.54 | Helped |
| Add the CMI objective by setting λ=0.15 rather than λ=0 (ablation) | MCMI teacher, ResNet-56→ResNet-20 · CIFAR-100 | student accuracy: 70.62 -> 70.84 | Helped |
| Increase λ from 0.15 to 0.25 (ablation) | MCMI teacher, ResNet-56→ResNet-20 · CIFAR-100 | student accuracy: 70.84 -> 70.66 | Hurt |
| Replace the MLL teacher with an MCMI teacher | Student in zero-shot KD · CIFAR-10, omitted classes | accuracy on omitted classes: 0% to as high as 84% | Helped |
| Replace the MLL teacher with an MCMI teacher | Student in few-shot KD · CIFAR-100, 75% training samples | Top-1 accuracy: +0.15 | Helped |
| Use MCMI rather than MLL teacher training | Teacher · CIFAR-100, teacher test set | accuracy | Hurt |
Surprising findings
- MCMI teachers have lower test accuracy than MLL teachers yet yield better students.
- Larger MLL teachers have small CMI values, which the authors argue explains why large teachers hurt students; MCMI mitigates this.
Limitations
- Teacher's test accuracy drops under MCMI, so the benefit depends on the KD setting rather than teacher accuracy.
- Requires a pre-trained MLL teacher and a tuned trade-off hyperparameter lambda per teacher.
Experimental setup
Datasets: CIFAR-100 ImageNet CIFAR-10 synthetic Gaussian dataset
Benchmarks: CIFAR-100 ImageNet CIFAR-10
Metrics: top-1 accuracy top-5 accuracy CMI LL
Baselines: KD AT PKT SP CC RKD VID CRD DKD ReviewKD HSAKD early-stopped teacher
Builds on: knowledge distillation CRD DKD Eigen-CAM
Closest papers by meaning
- Bayesian Knowledge Distillation: A Bayesian Perspective of Distillation with Uncertainty Quantification (ICML 2024 · similarity 0.90)
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.85)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.85)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.85)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.85)
- De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.