Research map / Federated learning and privacy / Personalized and heterogeneous FL
Efficient Federated Learning via Clients-to-Server Knowledge Distillation (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ClientsToServerKDFL, a federated learning method where clients act as multi-teacher distillation sources and the server is the student, to cut communication and computation costs.
Problem
Federated learning incurs high communication and client computation costs, and existing federated knowledge distillation methods demand heavy client-side computation.
Approach
Clients train locally and act as teachers whose logits are weighted by data quality (cross-entropy with true labels) in a multi-teacher KD scheme; the server acts as the student, distilling the aggregated knowledge and broadcasting it back, avoiding heavy client-side distillation.
Key results
- On MNIST, ClientsToServerKDFL trained in 146 s with 5.2E+06 tensors and 15.57M overhead, the fastest and lightest among compared methods (FedGen 338 s, FedKD 249 s).
- On CIFAR-10, it reported 2167 s training time versus 2472 s for FedNTD and 4286 s for FedGen.
- The paper claims about 50% parameter reduction and no accuracy loss, but accuracy numbers are only in supplementary material and not in the main text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ClientsToServerKDFL instead of FedNTD | CNN federated learning · MNIST | Time(s): 168 -> 146 | Helped |
| use ClientsToServerKDFL instead of FedProto | CNN federated learning · MNIST | Tensors: 5.2E+06 -> 5.2E+06 | No clear effect |
| use ClientsToServerKDFL instead of FedProto | CNN federated learning · MNIST | Overhead(M): 15.60 -> 15.57 | Helped |
| use ClientsToServerKDFL instead of FedNTD | CNN federated learning · CIFAR-10 | Time(s): 2472 -> 2167 | Helped |
| use ClientsToServerKDFL instead of FedProto | CNN federated learning · CIFAR-10 | Tensors: 2.9E+07 -> 2.8E+07 | Helped |
| use ClientsToServerKDFL instead of FedProto | CNN federated learning · CIFAR-10 | Overhead(M): 77.40 -> 77.19 | Helped |
Limitations
- Accuracy comparisons are not presented in the main text; claims of no accuracy loss rest on supplementary material.
- Evaluation limited to small CNN models and four standard image datasets with 10 clients.
- Theoretical convergence and privacy analyses are deferred to the supplement and their content is not verifiable here.
Experimental setup
Datasets: MNIST FMNIST CIFAR-10 CIFAR-100
Benchmarks: MNIST FMNIST CIFAR-10 CIFAR-100
Metrics: accuracy training time parameter count (tensors) memory overhead
Baselines: FedAvg FedGen FedKD FedProto FedNTD
Builds on: FedAvg FedKD FedGen FedProto FedNTD
Closest papers by meaning
- Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning (NeurIPS 2026 · similarity 0.90)
- One-shot Federated Learning via Synthetic Distiller-Distillate Communication (NeurIPS 2024 · similarity 0.89)
- Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning (NeurIPS 2025 · similarity 0.89)
- Knowledge Transfer via Compact Model in Federated Learning (Student Abstract) (AAAI 2024 · similarity 0.89)
- Communication-Efficient Federated Learning with Accelerated Client Gradient (CVPR 2024 · similarity 0.89)
- Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.