Research map / Optimization / Optimizers
Knowledge Distillation Performs Partial Variance Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use self-distillation with a properly tuned distillation weight | SGD linear models · MNIST and CIFAR-10 features | cross-entropy training loss | Helped |
| Use a better-trained teacher and increase the distillation weight | Self-distillation · MNIST linear classification | cross-entropy training loss | Helped |
| Correct the distillation update bias using the full-batch teacher gradient (ablation) | Self-distillation on CIFAR-10 features · CIFAR-10 features | cross-entropy training loss | Helped |
| Add self-distillation to randomly masked, fixed-sparsity updates | Compressed linear models · MNIST | cross-entropy training loss | Helped |
| Increase sparsity in compressed updates (ablation) | Self-distillation · MNIST | cross-entropy training loss | Mixed |
| Use bias-corrected self-distillation instead of biased self-distillation (ablation) | Linear models on CIFAR-10 features · CIFAR-10 features | gradient variance | No clear effect |
| Use knowledge distillation rather than plain SGD | Linear models on CIFAR-10 features · CIFAR-10 features | gradient variance | Helped |
Closest papers by meaning
- Bayesian Knowledge Distillation: A Bayesian Perspective of Distillation with Uncertainty Quantification (ICML 2024 · similarity 0.89)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.89)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.89)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.89)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.