Research map / Federated learning and privacy / Personalized and heterogeneous FL
Population Normalization for Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Population Normalization (PN), which learns BN-style population mean and variance as trainable parameters so normalization is consistent across heterogeneous clients and works with batch size 1, plus Noisy PN (NPN) that injects multiplicative noise to recover BN's generalization benefit at large batch sizes.
Problem
Batch normalization degrades in federated learning because client data are non-IID (so local BN statistics diverge across client models) and resource-limited clients can only use small mini-batches, which yield inaccurate statistic estimates.
Approach
Replace mini-batch mean and variance estimates with trainable population statistics constrained by the definitions of mean and variance, so normalization layers are homogeneous and aggregated like ordinary weights; for large batches, inject multiplicative Gaussian noise into the learned statistics to mimic BN's stochastic uncertainty and improve generalization.
Key results
- In FL with batch size 8 on CIFAR-10 (VGG-16, alpha=0.04), PN reaches 78.59% vs. 70.22% for BN; PN still trains at batch size 1 where BN is inapplicable (64.38% at alpha=0.04).
- With large batch size 2048 on CIFAR-10, NPN reaches 93.66% top-1 and loss 0.32, versus 93.13% and 0.39 for BN.
- Adding noise to batch renormalization improves CIFAR-10 top-1 from 93.21% to 93.60%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace mini-batch statistics with learned population statistics | VGG-16 · CIFAR-10, batch size 1 | accuracy: 71.83 -> 78.89 | Helped |
| Replace mini-batch statistics with learned population statistics | VGG-16 · CIFAR-100, batch size 1, α=0.04 | accuracy: NA -> 46.13 | Helped |
| Inject artificial noise into learned population statistics (ablation) | PN · CIFAR-10, large batch size | test loss: 0.41 -> 0.32 | Helped |
| Inject artificial noise into learned population statistics (ablation) | PN · CIFAR-100, large batch size | test loss: 1.85 -> 1.22 | Helped |
| Inject artificial noise into learned population statistics (ablation) | PN · CIFAR-10, large batch size | Top-1: 93.10 -> 93.66 | Helped |
| Use NPN instead of Batch Normalization | VGG-16 · CIFAR-10, large batch size | test loss: 0.39 -> 0.32 | Helped |
| Use NPN instead of Batch Normalization | VGG-16 · CIFAR-100, large batch size | test loss: 2.05 -> 1.22 | Helped |
| Increase noise variance from 0.001 to 0.5 (ablation) | NPN · CIFAR-10 | test loss | Hurt |
| Inject artificial noise into moving-average statistics (ablation) | Batch Renormalization · CIFAR-10 | Loss: 0.44 -> 0.29 | Helped |
| Use NPN instead of Batch Normalization | VGG-16 · CIFAR-10 supervised learning | accuracy: 92.31 -> 93.66 | Helped |
Surprising findings
- PN underperforms BN at large batch sizes due to lack of stochastic uncertainty, and injecting noise (NPN) is needed to close the gap.
- Higher-capacity networks do not necessarily yield better FL performance, contrary to standard training.
Limitations
- NPN is described as a normalization technique for FL rather than general; theoretical benefits of BN such as scale invariance are not analyzed.
- PN underperforms BN with very large batch sizes without noise injection.
- Experiments limited to CIFAR-10/100 image classification with VGG and ResNet models.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: top-1 accuracy top-5 accuracy test loss
Baselines: BN FedBN group normalization instance normalization without normalization batch renormalization
Builds on: FedAvg FedBN batch renormalization group normalization Dirichlet non-IID partitioning
Closest papers by meaning
- Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning (ICML 2025 · similarity 0.93)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.88)
- Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.87)
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization (ICLR 2024 · similarity 0.87)
- Improving Generalization in Federated Learning with Model-Data Mutual Information Regularization: A Posterior Inference Approach (NeurIPS 2024 · similarity 0.86)
- Federated Learning with Sample-level Client Drift Mitigation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.