Research map / Learning theory and algorithms / Kernels and regression theory
Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning
Hien Dang, Pratik Patil, Alessandro Rinaldo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add optimally mixed self-distillation (unconstrained xi) to the ridge teacher | Ridge regression · BlogFeedback, Communities and Crime, ResNet-18 CIFAR10 features | test squared prediction risk | Improved |
| Allow negative mixing weight xi in over-regularized regimes (ablation) | Self-distilled ridge · Real-data ridge path | test squared prediction risk | Improved |
| Add self-distillation to a ridge teacher under distribution shift | Ridge regression · Air Quality (out-of-distribution) | test squared prediction risk | Improved |
| Add optimal self-distillation to a ridge teacher at the globally optimal lambda | Ridge regression · Communities and Crime, CIFAR10 features | global minimum risk over lambda | No clear effect |
| Add optimal self-distillation to a ridge teacher at the globally optimal lambda | Ridge regression · BlogFeedback, Air Quality | global minimum risk over lambda | Improved |
| Use pure-distillation (xi=1) instead of optimal mixing (ablation) | Ridge regression · Real-data ridge path | test squared prediction risk | Hurt |
| Restrict xi to [0,1] instead of unconstrained (ablation) | Self-distilled ridge · Over-regularized teacher | test squared prediction risk | Hurt |
| Use recursive multi-round self-distillation with optimized xi each round | Self-distilled ridge · Air Quality, Communities and Crime | test squared prediction risk | Improved |
| Anchor later rounds to the original labels instead of recursive mixing (ablation) | Multi-round self-distillation · Air Quality, Communities and Crime | test squared prediction risk | Hurt |
| Distill on fresh unlabeled features instead of the same training features (ablation) | Self-distilled ridge · Isotropic setting n=200, p=100 | test squared prediction risk | Hurt |
| Estimate xi with one-shot GCV instead of the oracle | Self-distilled ridge · CIFAR10/CIFAR100 ResNet features, UCI data | SD risk estimate | Improved |
Closest papers by meaning
- High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws (ICLR 2025 · similarity 0.88)
- The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model (NeurIPS 2025 · similarity 0.87)
- Risk and cross validation in ridge regression with correlated samples (ICML 2025 · similarity 0.87)
- Optimal Ridge Regularization for Out-of-Distribution Prediction (ICML 2024 · similarity 0.87)
- Provable and Efficient Dataset Distillation for Kernel Ridge Regression (NeurIPS 2024 · similarity 0.87)
- Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
