Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation
Luyang Fang, Haoran Lu, Yongkai Chen, Wenxuan Zhong, Ping Ma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| tune smoothing parameters on a subsample and transfer (KCas) instead of full-sample GCV | smoothing spline ANOVA · Simulation 2 scenario 2 d=20, n=20000 | median time (min): 102.5 -> 8.1 | Improved |
| KCas instead of GCV on a random subsample (SUB) | smoothing spline density estimation · ESC density estimation | relative log-likelihood: 1.0369 -> 1.0475 | Improved |
| KCas instead of full-sample GCV | density estimation · ESC density estimation | relative log-likelihood | Improved |
| KCas instead of GAM-based GCV | density estimation · ESC density estimation | relative log-likelihood: 0.5514 -> 1.0475 | Improved |
| KCas instead of SKIP | nonparametric regression · SUSY regression | relative MSE: 0.8252 -> 0.7963 | Improved |
| KCas instead of GAM | nonparametric regression · OCUP regression | relative MSE: 4.1504 -> 0.9999 | Improved |
| KCas instead of GCV on a subsample (SUB) | nonparametric regression · WFRN regression | relative MSE: 1.3604 -> 1.0434 | Improved |
| KCas instead of SKIP when covariates are complex (d=6) | nonparametric regression · Simulation 2 scenario 1 d=6 | log RelMSE: SKIP median log-RelMSE greater than 1; KCas close to GCV | Improved |
| use KCas retuning-free transfer instead of full teacher grid search (Retune) | ResNet50 teacher on CIFAR-10 · CIFAR-10 (MobileNetV2 student) | time (hours): 121.1 -> 6.9 | Improved |
| use ResNet18 as student instead of MobileNetV2 | KCas hyperparameter transfer · CIFAR-10 teacher accuracy | accuracy | Improved |
| use square-root instead of linear learning-rate scaling | KCas deep transfer · CIFAR-10 | accuracy | No clear effect |
| KCas instead of cookbook hyperparameters | ResNet50 teacher on CIFAR-10 · CIFAR-10 | accuracy | Improved |
Closest papers by meaning
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.89)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.89)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.88)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.88)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.88)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
