AI Paper Atlas

Research map / Learning theory and algorithms / Kernels and regression theory

Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation

Kenneth Borup, Lars Nørvang Andersen

NeurIPS 2021 · Learning theory and algorithms › Kernels and regression theory · approach: High-dimensional asymptotic generalization · 4 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Include ground-truth targets with weight α > 0 during self-distillationKernel ridge regression self-distillation · Kernel ridge regressionregularization imposedHurt
Use α = 0 rather than a positive ground-truth weight (ablation)Kernel ridge regression self-distillation · Kernel ridge regressionsolution sparsificationImproved
Set α = 0 for every distillation step (ablation)Kernel ridge regression self-distillation · Kernel ridge regressionlimiting solution magnitudeHurt
Use a fixed positive α instead of α = 0 (ablation)Kernel ridge regression self-distillation · Kernel ridge regressionsolution stability across stepsImproved
Use step-wise optimal α instead of fixed α (ablation)Kernel ridge regression self-distillation · Illustrative RBF-kernel examplesolution change after step 2No clear effect
Use α ≤ 0.4 rather than α > 0.4 (ablation)ResNet-50 · CIFAR-10validation accuracyHurt
Use α > 0.4 rather than α ≤ 0.4 (ablation)ResNet-50 · CIFAR-10validation accuracyImproved
Choose α close to one rather than a moderate α (ablation)ResNet-50 · CIFAR-10validation accuracy gainsHurt
Estimate α at each step rather than perform a grid searchNeural-network self-distillation · CIFAR-10predictive performanceNo clear effect
Use the closed-form step-wise α estimate rather than grid searchNeural-network self-distillation · CIFAR-10model fits required: 2(τ-1)+1 fits compared to g(τ-1) +1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy