Research map / Learning theory and algorithms / Kernels and regression theory
A Kernel Perspective on Distillation-based Collaborative Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes distillation-based collaborative learning (FedMD) as nonparametric kernel regression, proving near-minimax optimality without sharing local data or models, and proposes DCL-NN with feature-kernel matching for neural networks.
Problem
Theoretical understanding of distillation-based collaborative learning (DCL) in massively distributed, statistically heterogeneous settings is lacking, and practical DCL methods with neural networks perform unsatisfactorily.
Approach
Model nonparametric FedMD as a kernel gradient-descent procedure on public inputs (DCL-KR) and prove a near-minimax rate whose prefactor is independent of the number of parties; for neural networks, align local last-hidden-layer feature kernels to an ensemble kernel via centered kernel alignment (DCL-NN).
Key results
- DCL-KR achieves performance comparable to central kernel regression models and outperforms DC-NY, DKRR-NY-CM and IED across 10-100 parties on Toy-1D and Toy-3D.
- DCL-NN achieves Toy-3D RMSE 0.079 vs 0.200 for FedMD and 0.288 for standalone; central is 0.041.
- DCL-NN achieves RotatedMNIST RMSE 0.227 vs 0.249 for FedMD and 0.317 for KT-pFL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DCL-NN instead of FedMD | Collaborative neural network regression · Toy-3D | RMSE: 0.200$\;\pm\;$0.008 -> 0.079$\;\pm\;$0.005 | Helped |
| Use DCL-NN instead of FedMD | Collaborative neural network regression · Energy | RMSE: 0.093$\;\pm\;$0.000 -> 0.087$\;\pm\;$0.001 | Helped |
| Use DCL-NN instead of FedMD | Collaborative neural network regression · RotatedMNIST | RMSE: 0.249$\;\pm\;$0.001 -> 0.227$\;\pm\;$0.003 | Helped |
| Use DCL-NN instead of FedMD | Collaborative neural network regression · UTKFace | RMSE: 0.151$\;\pm\;$0.004 -> 0.148$\;\pm\;$0.001 | Helped |
| Use DCL-NN instead of FedMD | Collaborative neural network regression · IMDB-WIKI | RMSE: 0.113$\pm$0.000 -> 0.110$\pm$0.000 | Helped |
Surprising findings
- Naive ensemble of local feature kernels performs significantly better than individual feature kernels.
- IED does not show a significant performance drop in massively distributed environments despite its theory assuming homogeneity.
Limitations
- Theory covers kernel gradient descent with fixed public inputs and early stopping; the neural network case lacks a formal guarantee.
- Requires public unlabeled data covering the input distribution; the distribution-mismatch factor B^r is removed only with more public inputs.
Experimental setup
Datasets: Toy-1D Toy-3D Energy RotatedMNIST UTKFace IMDB-WIKI
Benchmarks: Toy-1D Toy-3D Energy RotatedMNIST UTKFace IMDB-WIKI
Metrics: RMSE
Baselines: centralKRR centralKRGD DC-NY DKRR-NY-CM IED FedMD FedHeNN KT-pFL standalone
Builds on: FedMD FedAvg FedHeNN Nystrom kernel regression
Closest papers by meaning
- Collaborative Learning via Prediction Consensus (NeurIPS 2023 · similarity 0.89)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.85)
- Distillation Scaling Laws (ICML 2025 · similarity 0.85)
- Aequa: Fair Model Rewards in Collaborative Learning via Slimmable Networks (ICML 2025 · similarity 0.85)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.85)
- Collaborative Learning with Different Labeling Functions (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.