Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Knowledge Distillation of Uncertainty using Deep Latent Factor Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gaussian distillation, which compresses a deep ensemble into a single student DNN defining a deep latent factor Gaussian process (fit by EM with MMD-penalized initialization), and samples ensemble members from it to preserve uncertainty.
Problem
Knowledge distillation of deep ensembles into small models loses uncertainty, since smaller networks and weight sharing reduce variation; existing distribution distillation (Dirichlet) is unstable.
Approach
Treat teacher ensemble members as independent realizations of a Gaussian process whose mean and factor-loading functions are a student DNN (deep latent factor model), estimated stably via EM; at inference, sample ensemble members from the learned distribution to add variation.
Key results
- On CIFAR-10, DLF reaches 93.40% accuracy and 0.2246 NLL, versus 93.25% (LBE) and 92.87% (small-Ens), with a teacher ensemble at 94.24%.
- On CIFAR-100, DLF achieves NLL 0.8974 versus 1.0110 for LBE, though its ECE (9.45%) is worse than Proxy-EnD^2 (7.35%).
- On UCI regression, DLF is best or second best in most cases, e.g., Concrete RMSE 5.6047 vs 6.0558 for Hydra, using fewer parameters (1,612 vs 6,150 for Hydra on Concrete).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill the ensemble with Gaussian distillation (DLF) rather than LBE | CIFAR-10 ensemble · CIFAR-10 | accuracy (%): 93.25 (0.26) -> 93.40 (0.14) | Helped |
| distill the ensemble with Gaussian distillation (DLF) rather than small-Ens | CIFAR-10 ensemble · CIFAR-10 | NLL: 0.2377 (0.0019) -> 0.2246 (0.0023) | Helped |
| use DLF rather than Proxy-EnD² | CIFAR-10 ensemble · CIFAR-10 | ECE (%): 2.08 (0.19) -> 2.79 (0.20) | Hurt |
| distill the ensemble with Gaussian distillation (DLF) rather than LBE | CIFAR-100 ensemble · CIFAR-100 | accuracy (%): 79.58 (0.40) -> 79.68 (0.23) | Helped |
| use DLF rather than Proxy-EnD² | CIFAR-100 ensemble · CIFAR-100 | NLL: 1.2355 (0.0151) -> 0.8974 (0.0042) | Helped |
| use DLF rather than Proxy-EnD² | CIFAR-100 ensemble · CIFAR-100 | ECE (%): 7.35 (0.25) -> 9.45 (0.31) | Hurt |
| distill the teacher ensemble with DLF rather than small-Ens | DistilRoBERTa fine-tuning · RTE | ECE (%): 13.09 (0.0148) -> 9.74 (0.5050) | Helped |
| distill the teacher ensemble with DLF rather than small-Ens | DistilRoBERTa fine-tuning · MRPC | ECE (%): 9.84 (0.0164) -> 10.72 (0.0392) | Hurt |
| distill the teacher ensemble with DLF rather than small-Ens | DistilRoBERTa fine-tuning · WiC | accuracy (%): 65.02 (0.0062) -> 66.18 (0.0146) | Helped |
| distill the teacher ensemble with DLF rather than small-Ens | DistilRoBERTa fine-tuning · WiC | NLL: 0.7674 (0.0206) -> 0.7706 (0.0543) | Hurt |
| use DLF rather than Hydra or BE | regression ensemble · UCI regression benchmarks | RMSE | Helped |
| distill the ensemble with DLF rather than deterministic student methods | regression ensemble · Boston housing and Concrete | 95% coverage probability | Helped |
Surprising findings
- Dirichlet-based Proxy-EnD^2 and EDFM underperform one-to-one baselines on CIFAR-100 accuracy (67.62% and 64.17%) relative to small-Ens (79.29%).
Limitations
- Only deep ensembles studied; Bayesian DNNs left for future work
- Gaussian distillation is less effective than some baselines on certain metrics (e.g., ECE on CIFAR-100 vs Proxy-EnD^2; small-Ens NLL on Boston housing)
- Relies on design points (training/validation data) and on identifiability-sensitive initialization
Experimental setup
Datasets: UCI benchmark datasets (Boston housing, Concrete, Energy, Wine, Power, Kin8nm) CIFAR-10 CIFAR-100 RTE MRPC WiC CIFAR10-Flip
Benchmarks: UCI regression datasets CIFAR-10 CIFAR-100 RTE MRPC WiC CIFAR10-Flip
Metrics: RMSE NLL CRPS 95% coverage probability accuracy ECE
Baselines: small-Ens Hydra BE LBE Proxy-EnD^2 EDFM Dirichlet distillation
Builds on: Dirichlet distillation Hydra BatchEnsemble Latent Batch Ensemble LoRA
Closest papers by meaning
- Knowledge Distillation Performs Partial Variance Reduction (NeurIPS 2023 · similarity 0.88)
- Credal Ensemble Distillation for Uncertainty Quantification (AAAI 2026 · similarity 0.88)
- Bayesian Knowledge Distillation: A Bayesian Perspective of Distillation with Uncertainty Quantification (ICML 2024 · similarity 0.88)
- Discovering and Overcoming Limitations of Noise-engineered Data-free Knowledge Distillation (NeurIPS 2022 · similarity 0.88)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.85)
- Uncertainty Quantification with the Empirical Neural Tangent Kernel (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.