Research map / Federated learning and privacy / Personalized and heterogeneous FL
A Swiss Army Knife for Heterogeneous Federated Learning: Flexible Coupling via Trace Norm
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedSAK, a federated multi-task learning framework that splits client models into feature extractor and prediction head and couples the uploaded shared layers via a tensor trace norm regularizer, handling data, model, and task heterogeneity.
Problem
Federated learning under simultaneous data, model, and task heterogeneity, where most methods assume homogeneous architectures and tasks.
Approach
Treats each client as a task, lets clients choose which layers to share (full model, prediction head, or feature extractor), stacks shared layers into a p-way tensor on the server, and applies a Tucker-style tensor trace norm regularizer to induce a low-rank coupling across client models.
Key results
- On data-heterogeneous image classification (Table 1), FedSAK reaches 98.46% on HumA (10,6) and gains up to +13.2 points average over FedAvg across settings.
- Under model heterogeneity on PACS, FedSAK outperforms the best baseline FedGH by 5.98, 4.08, and 3.94 points for 4, 8, and 10 clients.
- Under task heterogeneity on Adience Faces with 2 clients, FedSAK reaches 88.78% gender accuracy and 65.13% age accuracy in the 1:1 setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use tensor trace norm for shared-layer knowledge transfer rather than no communication (Local) or weighted server aggregation (FedAvg) (ablation) | FedSAK · heterogeneous federated image classification; ablation setup | accuracy | Helped |
| Use tensor trace norm to couple client models | FedSAK · data heterogeneity across HumA, MNIST, CIFAR-10, and CIFAR-100 | accuracy | Helped |
| Use tensor trace norm on shared prediction heads | FedSAK · model heterogeneity on CIFAR-10 and PACS | accuracy: +5.98%, +4.08%, and +3.94% | Helped |
| Use tensor trace norm on shared feature extractors | FedSAK · Adience Faces task heterogeneity | accuracy | Mixed |
| Increase the task proportion of age-classification clients relative to gender-classification clients | FedSAK versus FedAvg-c · Adience Faces, 15 clients, task ratio 1:2, age (8 classes) | accuracy: 65.31 vs 65.19 | Hurt |
| Replace tensor trace norm coupling with Local training (no communication) (ablation) | FedSAK · heterogeneous federated image classification; ablation setup | accuracy | Hurt |
| Replace tensor trace norm coupling with FedAvg aggregation of uploaded shared layers (ablation) | FedSAK · heterogeneous federated image classification; ablation setup | accuracy | Hurt |
Surprising findings
- Optimal trace-norm coupling strength lambda is smaller for task heterogeneity and larger for data heterogeneity.
- FedAvg-c on task heterogeneity performs worse than local training for 2-class clients when task ratios are skewed.
Limitations
- Computational complexity of tensor trace norm grows significantly with model dimensionality.
- No statistical significance or error bars reported.
- Experiments limited to image classification with small CNNs and one ResNet18 run.
Experimental setup
Datasets: Human Activity MNIST CIFAR-10 CIFAR-100 PACS Adience Faces
Benchmarks: Human Activity MNIST CIFAR-10 CIFAR-100 PACS Adience Faces
Metrics: test accuracy running time per epoch communication overhead memory usage
Baselines: FedAvg FedProx MOON Per-FedAvg pFedMe Ditto GPFL FedAvgDBE FedMTL FedU FedMD FedProto
Builds on: FedMTL FedU MOCHA-style federated multi-task learning FedProto FedGH
Closest papers by meaning
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.89)
- Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning (ICLR 2024 · similarity 0.89)
- Federated Model Heterogeneous Matryoshka Representation Learning (NeurIPS 2024 · similarity 0.88)
- FedHCA2: Towards Hetero-Client Federated Multi-Task Learning (CVPR 2024 · similarity 0.88)
- Federated Learning with Sample-level Client Drift Mitigation (AAAI 2025 · similarity 0.88)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.