Research map / Federated learning and privacy / Personalized and heterogeneous FL
Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAKFL, a knowledge-distillation framework for federated learning across heterogeneous device prototypes that distills each prototype's ensemble separately with self-regularization and merges the results via task arithmetic.
Problem
Federated learning with device prototypes of widely varying model and dataset sizes, where existing ensemble knowledge distillation methods dilute informative logits from capable devices and use a single one-size-fits-all distillation target.
Approach
Treat distillation from each device prototype's ensemble as a separate task, self-regularize each student toward its averaged initialization, and integrate the resulting task vectors into the averaged parameters with merging coefficients (task arithmetic), extending task vectors from centralized learning to federated learning.
Key results
- On CIFAR-10 with Dir(0.3), TAKFL+Reg reaches 60.22% average accuracy versus 52.60% for FedET and 47.36% for FedAvg across S/M/L prototypes.
- On MNLI, TAKFL+Reg averages 63.49% versus 60.45% for FedDF; on SST-2 it averages 81.54% versus 79.82% for FedDF.
- FedET degrades strongly on NLP (SST-2 average 67.52% vs 72.13% for FedAvg), and the L prototype loses up to 10% vs FedAvg for prior KD methods on CIFAR-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TAKFL instead of FedDF | heterogeneous federated learning · CIFAR-10, Dir(0.3), average | top-1 accuracy: 49.92 -> 58.14 | Helped |
| use TAKFL instead of FedDF | heterogeneous federated learning · CIFAR-10, Dir(0.1), average | top-1 accuracy: 40.35 -> 42.62 | Helped |
| use TAKFL instead of FedDF | heterogeneous federated learning · CIFAR-100, Dir(0.3), average | top-1 accuracy: 24.38 -> 27.31 | Helped |
| use TAKFL instead of FedDF | heterogeneous federated learning · CIFAR-100, Dir(0.1), average | top-1 accuracy: 20.83 -> 24.50 | Helped |
| use TAKFL instead of FedDF | heterogeneous federated learning · MNLI, Dir(0.5), average | top-1 accuracy: 60.45 -> 63.25 | Helped |
| use TAKFL instead of FedDF | heterogeneous federated learning · SST-2, Dir(0.5), average | top-1 accuracy: 79.82 -> 81.28 | Helped |
| add self-regularization to TAKFL (ablation) | TAKFL · CIFAR-10, Dir(0.3), average | top-1 accuracy: 58.14 -> 60.22 | Helped |
| add self-regularization to TAKFL (ablation) | TAKFL · CIFAR-10, Dir(0.1), average | top-1 accuracy: 42.62 -> 44.62 | Helped |
| add self-regularization to TAKFL (ablation) | TAKFL · CIFAR-100, Dir(0.3), average | top-1 accuracy: 27.31 -> 29.15 | Helped |
| add self-regularization to TAKFL (ablation) | TAKFL · CIFAR-100, Dir(0.1), average | top-1 accuracy: 24.50 -> 24.94 | Helped |
| add self-regularization to TAKFL (ablation) | TAKFL · MNLI, Dir(0.5), average | top-1 accuracy: 63.25 -> 63.49 | Helped |
| add self-regularization to TAKFL (ablation) | TAKFL · SST-2, Dir(0.5), average | top-1 accuracy: 81.28 -> 81.54 | Helped |
Surprising findings
- Prior KD methods such as FedET can underperform vanilla FedAvg, especially for the large prototype, attributed to dilution of logits and poorly calibrated uncertainty estimates.
- Merging coefficients were set manually or via validation, so the adaptive integration is not learned automatically.
Limitations
- Not validated on physical devices or with extremely large models due to resource constraints.
- Merging coefficients are hyperparameters tuned manually or on held-out validation sets.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CINIC-10 TinyImageNet MNLI SST-2 MARC AG News ImageNet-100 SNLI Sentiment140
Benchmarks: CIFAR-10 CIFAR-100 MNLI SST-2
Metrics: top-1 classification accuracy
Baselines: FedAvg FedDF FedET
Builds on: FedDF FedET task arithmetic LwF
Closest papers by meaning
- An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning (CVPR 2024 · similarity 0.88)
- FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local Parameter Sharing (AAAI 2024 · similarity 0.88)
- Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning (NeurIPS 2025 · similarity 0.88)
- Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model (AAAI 2025 · similarity 0.88)
- Federated Learning via Input-Output Collaborative Distillation (AAAI 2024 · similarity 0.87)
- FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices Using a Computing Power-Aware Scheduler (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.