Research map / Federated learning and privacy / Personalized and heterogeneous FL
Exploiting Label Skews in Federated Learning with Model Concatenation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedConcat, which clusters clients by label distribution, runs FedAvg within each cluster, concatenates the cluster encoders, and trains a global classifier on top, supported by an information-bottleneck argument for concatenation over averaging.
Problem
Label skew (non-IID class distributions across clients) degrades federated learning accuracy because averaging local models that specialize in different classes produces a poor global model.
Approach
Replace parameter averaging with concatenation of cluster-level feature extractors: clients are grouped by label distribution (or inferred label distribution from model outputs on random inputs), FedAvg is run within each cluster, and the encoders are concatenated with a global linear classifier trained afterward with frozen encoders.
Key results
- FedConcat reaches 56.9% on CIFAR-10 with #C=2 versus 53.6% for FedAvg at the same communication cost, averaging about 4% improvement on CIFAR-10 across partitions.
- On CIFAR-100 with ResNet-50 (#C=2), FedConcat achieves 18.5% versus 8.4% for FedAvg, with an average gain of about 8% reported.
- FedConcat-ID, which infers label distributions from model outputs on random inputs without uploading label statistics, performs comparably to FedConcat (e.g., 56.5% vs 56.9% on CIFAR-10 #C=2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| concatenate clustered client encoders and train a global classifier instead of using model averaging | FedConcat · CIFAR-10 label-skew partitions | accuracy: 56.9%$\pm$0.2% vs 53.8%$\pm$0.8%; 62.0%$\pm$0.6% vs 59.1%$\pm$1.4%; 57.7%$\pm$0.4% vs 54.8%$\pm$0.3%; 64.2%$\pm$0.7% vs | Helped |
| concatenate clustered client encoders and train a global classifier instead of using model averaging | FedConcat · SVHN label-skew partitions | accuracy: 83.4%$\pm$1.4%; 86.0%$\pm$0.9%; 83.2%$\pm$0.9%; 87.5%$\pm$0.1% | Mixed |
| concatenate clustered client encoders and train a global classifier instead of using model averaging | FedConcat · FMNIST label-skew partitions | accuracy: 84.4%$\pm$0.6%; 87.1%$\pm$0.2%; 84.5%$\pm$0.1%; 87.7%$\pm$0.1% | Mixed |
| concatenate clustered client encoders and train a global classifier instead of using model averaging | FedConcat · CIFAR-100 label-skew partitions | accuracy: 18.5%$\pm$0.7%; 34.4%$\pm$1.1%; 61.2%$\pm$0.4%; 66.3%$\pm$0.1% | Helped |
| concatenate clustered client encoders and train a global classifier instead of using model averaging | FedConcat · Tiny-ImageNet label-skew partitions | accuracy: 4.3%$\pm$0.1%; 11.7%$\pm$0.3%; 43.1%$\pm$0.2%; 44.3%$\pm$0.1% | Helped |
| infer client label distributions from local models rather than upload label distributions | FedConcat-ID · CIFAR-10, SVHN, and FMNIST label-skew partitions | accuracy: CIFAR-10: 56.5%$\pm$2.6%; SVHN: 83.2%$\pm$1.9%; FMNIST: 83.0%$\pm$2.0% | Mixed |
| use clustering before concatenating client encoders instead of concatenating all client models (ablation) | FedConcat · CIFAR-10 | test accuracy | Helped |
| increase the number of clusters too much (ablation) | FedConcat · CIFAR-10 and FMNIST | test accuracy | Hurt |
| use a small number of clusters (ablation) | FedConcat · CIFAR-10 and FMNIST | test accuracy | No clear effect |
| add differential privacy noise to uploaded label distributions (DP-FedConcat, epsilon=2.5) (ablation) | FedConcat · CIFAR-10 #C=2 | accuracy: 54.7%$\pm$0.6% vs 53.6%$\pm$0.8% | Helped |
| add differential privacy noise to uploaded label distributions (DP-FedConcat, epsilon=2.5) (ablation) | FedConcat · FMNIST Dir(0.1) and Dir(0.5) | accuracy: 82.8%$\pm$3.3% vs 85.1%$\pm$0.5%; 87.2%$\pm$0.1% vs 87.5%$\pm$0.3% | Hurt |
| concatenate encoders rather than average them | FedConcat encoder · combined FMNIST data from two #C=2 clients | classification loss: 0.1072 -> 0.0493 | Helped |
Surprising findings
- Under extreme label skew, averaging local models causes accuracy to drop well below the individual local models, as shown in a two-client CIFAR-10 example.
- On SVHN and FMNIST with Dirichlet partitions, FedConcat offers only comparable accuracy to baselines, since label skew is slight and baselines are near centralized accuracy.
Limitations
- Requires client label distributions (privacy concern) unless FedConcat-ID is used; FedConcat-ID's inference is an approximation.
- Does not work for clients with only one label (#C=1) according to the authors' experiments.
- Needs cluster-count selection (elbow method), and the final model grows with the number of clusters; ResNet-50 experiments required heuristic adaptations such as increased weight decay and cluster rebalancing.
Experimental setup
Datasets: CIFAR-10 FMNIST SVHN CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 SVHN FMNIST CIFAR-100 Tiny-ImageNet
Metrics: test accuracy
Baselines: FedAvg FedProx MOON FedRS FedLC IFCA FedSoft FeSEM SCAFFOLD
Builds on: FedAvg information bottleneck K-means
Closest papers by meaning
- Exploring Vacant Classes in Label-Skewed Federated Learning (AAAI 2025 · similarity 0.91)
- FedFixer: Mitigating Heterogeneous Label Noise in Federated Learning (AAAI 2024 · similarity 0.86)
- Federated Partial Label Learning with Local-Adaptive Augmentation and Regularization (AAAI 2024 · similarity 0.86)
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization (ICLR 2024 · similarity 0.86)
- Language-Guided Transformer for Federated Multi-Label Classification (AAAI 2024 · similarity 0.86)
- A Simple Data Augmentation for Feature Distribution Skewed Federated Learning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.