Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Bayesian Knowledge Distillation: A Bayesian Perspective of Distillation with Uncertainty Quantification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bayesian Knowledge Distillation (BKD), which places a teacher-informed Dirichlet prior on student parameters so KD loss minimization equals posterior mode estimation, and samples the posterior with SGLD for uncertainty quantification and credible intervals.
Problem
Knowledge distillation lacks a clear statistical explanation and provides no principled way to quantify the uncertainty of the distilled student model's predictions.
Approach
Treat the cross-entropy to labels as a likelihood and the teacher's probabilities as a Dirichlet teacher-informed prior over student parameters; the KD loss then equals the negative log posterior, so KD = posterior mode, and SGLD sampling of the posterior gives mean-deviance uncertainty and credible intervals.
Key results
- BKD accuracy on CIFAR-100 0.842 vs original KD 0.841 and teacher 0.929; on CIFAR-10 0.964 vs KD 0.963 with teacher 0.989.
- On perturbed MNIST, original KD becomes overconfident at high perturbation levels while BKD mean deviance rises steadily (e.g., 1.46 at gamma=1 vs KD 0.38).
- Simulation scenario 4 MAE drops from 0.044 (orig KD) to 0.037 (BKD); 95% credible intervals achieve coverage near 0.95 on CIFAR-100 test subsets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill the teacher with BKD instead of Orig KD | student model · Fashion MNIST | accuracy: 0.902 -> 0.905 | Helped |
| distill the teacher with BKD instead of Orig KD | student model · CIFAR-10 | accuracy: 0.963 -> 0.964 | Helped |
| distill the teacher with BKD instead of Orig KD | student model · CIFAR-100 | accuracy: 0.841 -> 0.842 | Helped |
| distill the teacher with BKD instead of Orig KD | student model · MNIST | accuracy: 0.986 -> 0.984 | No clear effect |
| use BKD instead of the teacher model | student model · Fashion MNIST | accuracy: 0.901 -> 0.905 | Helped |
| use BKD instead of the teacher model | student model · CIFAR-100 | accuracy: 0.929 -> 0.842 | Hurt |
| use BKD instead of the teacher model | student model · Scenario 4 | accuracy: 0.838 -> 0.870 | Helped |
| use BKD instead of Orig KD | student model · Scenario 1 | MAE: 0.027 -> 0.023 | Helped |
| use BKD instead of Orig KD | student model · Scenario 2 | accuracy: 0.896 -> 0.901 | Helped |
| use BKD instead of Orig KD | student model · Scenario 3 | accuracy: 0.858 -> 0.868 | Helped |
| use BKD instead of Orig KD | student model · Scenario 4 | accuracy: 0.862 -> 0.870 | Helped |
| replace the Dirichlet prior with a continuous categorical prior (ablation) | BKD on MNIST · MNIST | accuracy: 0.980 -> 0.979 | No clear effect |
Surprising findings
- Original KD's predictive deviance decreases at high perturbation levels, indicating overconfident wrong predictions, whereas BKD's uncertainty keeps increasing.
Limitations
- Uncertainty is measured via deviance and Dirichlet prior choice is ad hoc; alternative priors (continuous categorical) change uncertainty behavior.
- Relies on the teacher's predicted probabilities for the training set; experiments use relatively small student models and mostly classification benchmarks.
- Accuracy gains over original KD are marginal (e.g., 0.001 to 0.002 on CIFAR datasets).
Experimental setup
Datasets: MNIST Fashion MNIST CIFAR-10 CIFAR-100 synthetic simulation scenarios
Benchmarks: MNIST Fashion MNIST CIFAR-10 CIFAR-100
Metrics: classification accuracy mean deviance coverage rate of credible intervals MAE of class probability
Baselines: Teacher model Original KD KD + BNN KD + Dropout
Builds on: knowledge distillation stochastic gradient Langevin dynamics Bayesian neural networks Monte Carlo dropout Bayesian Dark Knowledge
Closest papers by meaning
- Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information (ICLR 2024 · similarity 0.90)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.88)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.88)
- Knowledge Distillation of Uncertainty using Deep Latent Factor Model (NeurIPS 2025 · similarity 0.88)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.88)
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.