Research map / Federated learning and privacy / Personalized and heterogeneous FL
Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedFD, a feature-based ensemble distillation method for model-heterogeneous federated learning that aligns client features via orthogonal (Stiefel-manifold) projection layers re-parameterized by skew-symmetric matrices.
Problem
Logit-based ensemble distillation in model-heterogeneous federated learning (Hetero-FL) is unstable and fails to align features from heterogeneous client architectures, producing biased teacher knowledge.
Approach
Replace logit distillation with feature distillation: the server keeps one projection layer per client feature-dimension group, parameterized as a truncated matrix exponential of a skew-symmetric matrix so it stays orthogonal, and aligns server features to averaged client features with KL divergence.
Key results
- FedFD achieves the best global accuracy in all nine dataset/alpha settings, e.g. 90.06% on CIFAR-10 (alpha=10.0) versus 89.26% for DaFKD.
- The paper reports improvements of up to 16.09% in global model accuracy over baselines.
- Ablation shows removing orthogonal projection or hierarchical feature alignment drops accuracy by 0.63%-2.43%, with orthogonal projection contributing more.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedFD feature distillation instead of the strongest listed baseline | Global model · CIFAR-10, α=10.0 | test accuracy: 89.26 -> 90.06 | Helped |
| Use FedFD feature distillation instead of the strongest listed baseline | Global model · CIFAR-10, α=1.0 | test accuracy: 87.22 -> 89.64 | Helped |
| Use FedFD feature distillation instead of the strongest listed baseline | Global model · CIFAR-10, α=0.1 | test accuracy: 80.07 -> 82.74 | Helped |
| Use FedFD feature distillation instead of the strongest listed baseline | Global model · CIFAR-100, α=10.0 | test accuracy: 59.53 -> 61.07 | Helped |
| Use FedFD feature distillation instead of the strongest listed baseline | Global model · CIFAR-100, α=1.0 | test accuracy: 59.09 -> 60.86 | Helped |
| Use FedFD feature distillation instead of the strongest listed baseline | Global model · CIFAR-100, α=0.1 | test accuracy: 57.33 -> 59.24 | Helped |
| Use FedFD feature distillation instead of the strongest listed baseline | Global model · Tiny-ImageNet, α=10.0 | test accuracy: 36.86 -> 40.27 | Helped |
| Use FedFD feature distillation instead of the strongest listed baseline | Global model · Tiny-ImageNet, α=1.0 | test accuracy: 32.57 -> 34.24 | Helped |
| Use FedFD feature distillation instead of the strongest listed baseline | Global model · Tiny-ImageNet, α=0.1 | test accuracy: 27.53 -> 29.09 | Helped |
| Add orthogonal projection (ablation) | FedFD · Test accuracy across evaluated settings | test accuracy | Helped |
| Add hierarchical feature alignment (ablation) | FedFD · Test accuracy across evaluated settings | test accuracy | Helped |
| Add both hierarchical feature alignment and orthogonal projection (ablation) | FedFD · Test accuracy across evaluated settings | test accuracy: 0.63%~2.43% | Helped |
Surprising findings
- Logit-based ensemble distillation is unstable under model heterogeneity even though it works well in homogeneous settings.
Limitations
- Ablation gains from hierarchical feature alignment are small, and the authors attribute this partly to experiment scale.
- Experiments use only image classification with a ResNet-18 server model and a fixed set of 10-level width-scaled client variants.
- Each experiment is run only twice, limiting statistical confidence.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: test accuracy (global model) test accuracy (local models) communication rounds to target accuracy
Baselines: HeteroFL MOON FedFusion FedGen DaFKD FedMD MSFKD FedGD
Builds on: HeteroFL FedMD FedAvg MOON FedGen
Closest papers by meaning
- Efficient Federated Learning via Clients-to-Server Knowledge Distillation (Student Abstract) (AAAI 2025 · similarity 0.89)
- Federated Model Heterogeneous Matryoshka Representation Learning (NeurIPS 2024 · similarity 0.88)
- An Aggregation-Free Federated Learning for Tackling Data Heterogeneity (CVPR 2024 · similarity 0.88)
- Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration (NeurIPS 2024 · similarity 0.88)
- HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning (NeurIPS 2024 · similarity 0.88)
- Fake It Till Make It: Federated Learning with Consensus-Oriented Generation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.