Research map / Federated learning and privacy / Personalized and heterogeneous FL

FedWon: Triumphing Multi-domain Federated Learning Without Normalization

ICLR 2024 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated learning with pretrained models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FedWon, which removes all normalization layers from federated networks and uses scaled weight standardization convolutions (WSConv) with AGC to handle multi-domain non-i.i.d. federated learning.

Problem

Federated learning with multi-domain clients (each client's data from a different feature distribution) where batch normalization statistics mismatch across domains, hurting global model performance and making personalized BN methods unsuitable for cross-device FL.

Approach

Remove all normalization layers in FL and reparameterize convolutions with scaled weight standardization (WSConv), so the model has no batch-dependent statistics to aggregate, combined with FedAvg protocols and adaptive gradient clipping.

Key results

• On Office-Caltech-10 with batch size 1, FedWon reaches 66.7/55.1/96.9/89.8 across Amazon/Caltech/DSLR/Webcam, while FedAvg+GN and FedAvg+LN are lower.
• On Digits-Five, FedWon averages 88.5% versus 88.4% for FedBN and 86.5% for FedAvg; on DomainNet FedWon averages 51.1% versus 46.8% for FedBN.
• On Office-Caltech-10 DSLR, FedWon reaches 95.3% vs 85.4% for FedBN, a gain of nearly 10 points in the main table; the paper reports over 17% gains on some domains in cross-silo settings.

Measured effects reported in the paper

Surprising findings

• Simply removing BN with FedAvg (without WSConv) yields much worse results than SingleSet, showing that normalization-free FL is not trivially effective without the reparameterization.
• FedAvg+GN and FedAvg+LN perform similarly to or worse than FedAvg on ResNet-20, unlike on AlexNet.

Limitations

• Gains are sometimes small or negative relative to FedBN on some domains (e.g., Digits-Five MNIST-M, Quickdraw), and FedBN is outperforming on a few domains.
• Experiments use small subsets of datasets and few clients, and multiple results rely on mean of three runs.
• Skewed label distribution evaluation is limited to CIFAR-10 with MobileNetV2 and a single backbone.

Experimental setup

Datasets: Digits-Five Office-Caltech-10 DomainNet CIFAR-10 Fed-ISIC2019 HAM10000 ISIC2019

Benchmarks: Digits-Five Office-Caltech-10 DomainNet CIFAR-10 (label skew) Fed-ISIC2019

Metrics: testing accuracy average accuracy

Baselines: FedAvg FedProx FedBN SiloBN FixBN FedAvg+GN FedAvg+LN SingleSet PartialFed FedAvg w/o BN

Builds on: FedAvg FedBN Scaled Weight Standardization (NFNet) Adaptive Gradient Clipping SiloBN FixBN

Closest papers by meaning

• A Simple Data Augmentation for Feature Distribution Skewed Federated Learning (CVPR 2025 · similarity 0.89)
• Federated Learning with Domain Shift Eraser (CVPR 2025 · similarity 0.89)
• Little Is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning (AAAI 2025 · similarity 0.89)
• FLUX: Efficient Descriptor-Driven Clustered Federated Learning under Arbitrary Distribution Shifts (NeurIPS 2025 · similarity 0.89)
• Benchmarking Algorithms for Federated Domain Generalization (ICLR 2024 · similarity 0.88)
• (FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning (NeurIPS 2024 · similarity 0.88)