Research map / Learning theory and algorithms / Kernels and regression theory
Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
Kenneth Borup, Lars Nørvang Andersen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Include ground-truth targets with weight α > 0 during self-distillation | Kernel ridge regression self-distillation · Kernel ridge regression | regularization imposed | Hurt |
| Use α = 0 rather than a positive ground-truth weight (ablation) | Kernel ridge regression self-distillation · Kernel ridge regression | solution sparsification | Improved |
| Set α = 0 for every distillation step (ablation) | Kernel ridge regression self-distillation · Kernel ridge regression | limiting solution magnitude | Hurt |
| Use a fixed positive α instead of α = 0 (ablation) | Kernel ridge regression self-distillation · Kernel ridge regression | solution stability across steps | Improved |
| Use step-wise optimal α instead of fixed α (ablation) | Kernel ridge regression self-distillation · Illustrative RBF-kernel example | solution change after step 2 | No clear effect |
| Use α ≤ 0.4 rather than α > 0.4 (ablation) | ResNet-50 · CIFAR-10 | validation accuracy | Hurt |
| Use α > 0.4 rather than α ≤ 0.4 (ablation) | ResNet-50 · CIFAR-10 | validation accuracy | Improved |
| Choose α close to one rather than a moderate α (ablation) | ResNet-50 · CIFAR-10 | validation accuracy gains | Hurt |
| Estimate α at each step rather than perform a grid search | Neural-network self-distillation · CIFAR-10 | predictive performance | No clear effect |
| Use the closed-form step-wise α estimate rather than grid search | Neural-network self-distillation · CIFAR-10 | model fits required: 2(τ-1)+1 fits compared to g(τ-1) +1 | Improved |
Closest papers by meaning
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.87)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.86)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.86)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.85)
- Provable and Efficient Dataset Distillation for Kernel Ridge Regression (NeurIPS 2024 · similarity 0.85)
- The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
