Research map / Federated learning and privacy / Personalized and heterogeneous FL

FedGMKD: An Efficient Prototype Federated Learning Framework through Knowledge Distillation and Discrepancy-Aware Aggregation

NeurIPS 2024 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Non-IID federated learning · 15 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FedGMKD, a prototype-based federated learning framework that uses GMM-based cluster knowledge fusion for distillation and discrepancy-aware (KL-weighted) aggregation, without public data or server generators, evaluated on SVHN, CIFAR-10, CIFAR-100 with ResNet-18.

Problem

Data heterogeneity (Non-IID) across clients in federated learning hurts convergence and both local and global accuracy; existing knowledge-distillation-based personalized FL often needs public data or server-side generators.

Approach

Clients summarize their data with GMM-derived per-class prototype features and soft predictions (cluster knowledge fusion); the server aggregates them with weights combining data volume and a KL-divergence-based discrepancy (quality) measure, and clients regularize local training toward the global prototypes and soft predictions.

Key results

• On CIFAR-10 with 50 clients (beta=0.5), FedGMKD reaches 65.69% local and 60.31% global accuracy vs. 61.51% and 51.53% for FedAvg.
• On SVHN with 10 clients, FedGMKD reaches 86.26% local and 82.64% global accuracy, above FedProto (85.58%, 81.34%).
• On CIFAR-100 with 50 clients, FedGMKD reaches 23.57% local and 24.63% global accuracy vs. 22.60% and 22.86% for FjORD.

Measured effects reported in the paper

Surprising findings

• With ResNet-50 on CIFAR-10, FedGMKD's global accuracy (50.48%) is far below its ResNet-18 result (60.31%), which the authors attribute to limited communication rounds.
• In the beta=0.2 SVHN ablation, CKF-only gives higher local accuracy than the full model's partner DAT-only variant but lower global accuracy, and FedProto's global accuracy on CIFAR-10 beta=0.2 is reported at 37.36%.

Limitations

• Computational overhead over FedAvg is acknowledged as moderate, and the authors list efficiency and scalability as future work.
• Performance drops with ResNet-50 relative to ResNet-18 under the fixed round budget, so deeper models are not handled well.
• Key hyperparameters (lambda, gamma, a, b, temperature) are tuned and sensitive, and only a single small set of communication rounds (50) is used.

Experimental setup

Datasets: SVHN CIFAR-10 CIFAR-100 IMDB

Benchmarks: SVHN CIFAR-10 CIFAR-100 IMDB

Metrics: local accuracy global accuracy wall-clock time per round

Baselines: FedAvg FedProx MOON FedMD FedGen FedProto FPL FjORD

Builds on: FedHKD FedProto FedAvg knowledge distillation

Closest papers by meaning

• Efficient Federated Learning via Clients-to-Server Knowledge Distillation (Student Abstract) (AAAI 2025 · similarity 0.88)
• Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning (NeurIPS 2025 · similarity 0.88)
• FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.87)
• Bridging Model Heterogeneity in Federated Learning via Uncertainty-based Asymmetrical Reciprocity Learning (ICML 2024 · similarity 0.87)
• Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domains (NeurIPS 2024 · similarity 0.87)
• HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning (NeurIPS 2024 · similarity 0.87)