Research map / Federated learning and privacy / Personalized and heterogeneous FL

FedHCA2: Towards Hetero-Client Federated Multi-Task Learning

CVPR 2024 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Non-IID federated learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Hetero-Client Federated Multi-Task Learning (HC-FMTL) and the FedHCA2 framework, which aggregates heterogeneous clients' encoders and decoders separately using conflict-averse encoder aggregation, layer-wise cross-attention decoder aggregation, and learnable per-client aggregation weights.

Problem

Federated multi-task learning assumes clients share identical model architectures and task sets, which fails when clients have different task setups and thus incongruent multi-task models, on top of data and task heterogeneity.

Approach

Server disassembles client models into shared encoders and task decoders; encoders are aggregated with a CAGrad-style conflict-averse update derived from a theoretical MTL-FL equivalence, decoders are aggregated per layer with cross-attention across all decoders, and learnable Hyper Aggregation Weights personalize each client's update.

Key results

• On HC-FMTL scenario with five single-task clients (PASCAL-Context) and one multi-task client (NYUD-v2), FedHCA2 achieves Δm% of 2.18 versus 1.47 for FedAMP and -0.46 for MaT-FL (local baseline 0.00).
• In the reverse scenario (four single-task NYUD-v2 clients, one multi-task PASCAL-Context client), FedHCA2 obtains Δm% of 0.75, best among compared methods (FedPer -0.16, MaT-FL -0.16).
• Ablation shows encoder aggregation alone gives Δm% 1.89 and decoder aggregation alone 1.80; combined gives 2.18.

Measured effects reported in the paper

Surprising findings

• Several standard FL baselines (FedAvg, FedProx) substantially underperform local training, with Δm% around -8 to -12, showing naive aggregation harms multi-task dense prediction under heterogeneity.
• Performance of FedHCA2 improves as the number of clients per task scales, while most other methods degrade.

Limitations

• Evaluation limited to two dense-prediction benchmarks (PASCAL-Context and NYUD-v2) with a single shared Swin-T architecture, so model heterogeneity is not explored.
• Gains over local baseline are modest (about 1-2% Δm), and some individual metrics are worse than baselines.
• Theoretical link between MTL and FL relies on simplifying assumptions (one mini-batch, identical initialization).

Experimental setup

Datasets: PASCAL-Context NYUD-v2

Benchmarks: PASCAL-Context NYUD-v2

Metrics: mIoU maxF mErr odsF RMSE average per-task performance drop (Δm%)

Baselines: FedAvg FedProx FedPer Ditto FedBABU FedAMP FedSTA MaT-FL Local training

Builds on: CAGrad MaT-FL FedAvg MTI-Net / cross-task interaction (prior MTL task interaction methods)

Closest papers by meaning

• A Swiss Army Knife for Heterogeneous Federated Learning: Flexible Coupling via Trace Norm (NeurIPS 2024 · similarity 0.88)
• FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation (ICML 2025 · similarity 0.87)
• FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local Parameter Sharing (AAAI 2024 · similarity 0.87)
• FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.87)
• Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model (AAAI 2025 · similarity 0.87)
• FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning (AAAI 2024 · similarity 0.87)