Research map / Federated learning and privacy / Personalized and heterogeneous FL
Knowledge Transfer via Compact Model in Federated Learning (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a federated learning framework where clients pre-train large local models and transfer knowledge to a compact same-architecture global model via prediction loss and KL alignment of softmax outputs, reducing communication overhead.
Problem
Communication overhead in federated learning caused by frequent exchange of large global models, while preserving accuracy and client privacy.
Approach
Clients pre-train large local models; a compact global model (same architecture, smaller) is communicated and updated on clients using label loss plus KL divergence to the local model's softmax outputs, then aggregated by dataset-size-weighted averaging.
Key results
- On FMNIST (alpha=0.2), the method reaches 89.21% accuracy versus 83.45% for FedAvg, with 90.1 Mb overhead versus 100 Mb.
- On CIFAR10 (alpha=0.2), the method reaches 72.16% accuracy versus 71.75% for FedGen, with 4.01E+04 Mb overhead versus 6.45E+04 Mb for FedGen and 7.70E+04 Mb for FedAvg.
- Training time on CIFAR10 is 7.51E+03 s versus 7.91E+04 s for FedGen, per the paper's Table 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use compact-model knowledge transfer instead of FEDGEN | Federated learning · FMNIST (α=0.2) | accuracy: 88.54 -> 89.21 | Helped |
| Use compact-model knowledge transfer instead of PrunFL | Federated learning · FMNIST (α=0.2) | communication overhead: 9.55E+01 -> 9.01E+01 | Helped |
| Use compact-model knowledge transfer instead of PrunFL | Federated learning · FMNIST (α=0.2) | training time: 1.35E+03 -> 7.24E+02 | Helped |
| Use compact-model knowledge transfer instead of FEDGEN | Federated learning · CIFAR10 (α=0.2) | accuracy: 71.75 -> 72.16 | Helped |
| Use compact-model knowledge transfer instead of FEDGEN | Federated learning · CIFAR10 (α=0.2) | communication overhead: 6.45E+04 -> 4.01E+04 | Helped |
| Use compact-model knowledge transfer instead of PrunFL | Federated learning · CIFAR10 (α=0.2) | training time: 8.91E+03 -> 7.51E+03 | Helped |
Limitations
- Efficacy depends on robust client-side data, as stated in the conclusion.
- Evaluated only on two small image datasets with CNN and VGG9 models.
- Privacy claims are not formally analyzed.
Experimental setup
Datasets: FMNIST CIFAR10
Benchmarks: FMNIST CIFAR10
Metrics: accuracy communication overhead (Mb) training time (s)
Baselines: FedAvg PrunFL FedGen
Builds on: FedAvg knowledge distillation
Closest papers by meaning
- Efficient Federated Learning via Clients-to-Server Knowledge Distillation (Student Abstract) (AAAI 2025 · similarity 0.89)
- An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning (CVPR 2024 · similarity 0.88)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.87)
- Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.87)
- Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model (AAAI 2025 · similarity 0.87)
- Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.