Research map / Label-efficient and robust learning / Knowledge distillation
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| average the outputs of ten independently trained networks | ResNet-34 · CIFAR-100 | test accuracy: $5\%$ | Helped |
| knowledge-distill the ensemble into a single model of the same architecture | single network · deep learning image classification | test accuracy | Helped |
| distill an ensemble into a random feature mapping (NTK) model | random feature / NTK models · CIFAR | test accuracy | No clear effect |
| ensemble random feature mappings (NTK) instead of deep networks | NTK · CIFAR | test accuracy | No clear effect |
| ensemble models on Gaussian-like synthetic inputs | neural networks · synthetic Gaussian-like data | test accuracy | No clear effect |
| train directly to learn the average of individual networks instead of averaging trained networks | neural networks · deep learning | test accuracy | Hurt |
| ensemble after deleting 7/8 of channels in an intermediate ResNet layer (ablation) | ResNet · CIFAR | ensemble test accuracy | No clear effect |
| further ensemble models that were distilled from an ensemble | distilled single models · deep learning | test accuracy | No clear effect |
Closest papers by meaning
- Diversity Matters When Learning From Ensembles (NeurIPS 2021 · similarity 0.89)
- Knowledge Distillation by On-the-Fly Native Ensemble (NeurIPS 2018 · similarity 0.89)
- Ensemble Distribution Distillation via Flow Matching (ICML 2025 · similarity 0.87)
- Knowledge Distillation of Uncertainty using Deep Latent Factor Model (NeurIPS 2025 · similarity 0.85)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.85)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.