Research map / Federated learning and privacy / Personalized and heterogeneous FL
Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hybrid Batch Normalisation (HBN), which separates BN statistic updates from learnable-parameter updates to obtain unbiased global statistics in federated learning and learnably mixes batch and global statistics per client.
Problem
Standard batch normalisation degrades in federated learning with non-IID client data and small batch sizes, because local batch statistics mismatch global statistics used at evaluation time.
Approach
Decouple BN running-statistic updates from SGD by computing local statistics under the frozen previous global model, aggregate them into unbiased global mean/variance on the server, and at each client mix current-batch statistics with these global statistics via a learnable per-channel hybrid factor (sigmoid-weighted), with the global statistics frozen during gradient updates.
Key results
- HBN reaches 78.22% top-1 on CIFAR-10 (phi=0.6, B=4, Simple-CNN) vs 75.82% for BN and 77.34% for FixBN.
- On CIFAR-100 with batch size 2 to 32, HBN's accuracy drops by 2.19 points from B=32 to B=2, versus 5.74 points for BN.
- HBN improves ResNet-50 on CIFAR-100 (phi=0.1, B=64) to 60.74% vs 54.97% with BN and 55.37% with GN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HBN instead of the strongest listed baseline | FedAvg + Simple-CNN · CIFAR-10, φ=0.6/0.3/0.1, B=4 | top-1 test accuracy (%): 78.22 vs 77.34; 76.53 vs 74.51; 71.56 vs 70.36 | Helped |
| use HBN instead of the strongest listed baseline | FedAvg + Simple-CNN · CIFAR-100, φ=0.6/0.3/0.1, B=4 | top-1 test accuracy (%): 49.88 vs 47.11; 48.93 vs 46.33; 46.52 vs 43.80 | Helped |
| use HBN instead of FixBN | FedAvg + ResNet-18 · Tiny-ImageNet, φ=0.1/0.05, B=64 | top-1 test accuracy (%): 25.59 vs 24.71; 24.69 vs 23.42 | Helped |
| use HBN instead of BN | FedAvg + Simple-CNN · CIFAR-100, φ=0.6, B=32 to B=2 | test accuracy (%): 51.05 vs 49.63; 48.86 vs 43.89 | Helped |
| add HBN normalisation | FedAvg and other tested FL methods · CIFAR-10, φ=0.6, B=4 | test accuracy (%): FedAvg 75.82 -> 78.22; FedProx 76.08 -> 78.44; FedAdam 75.67 -> 78.17; Scafflod 77.63 -> 78.65; Moon 75.77 -> 78.48; Fed | Helped |
| replace BN or GN with HBN | MobileNetV2, ResNet-18/50, VGG-11/19 · CIFAR-100, φ=0.1, B=64 | test accuracy (%): MobileNetV2: 40.19/41.50 -> 44.40; ResNet-18: 51.51/52.38 -> 55.69; ResNet-50: 54.97/55.37 -> 60.74; VGG-11: 62.74/59.47 | Helped |
| replace separated local-statistics calculation with moving average (ablation) | FedAvg + HBN + Simple-CNN · CIFAR-10, φ=0.6, B=4 | test accuracy (%): 78.22 -> 72.25 | Hurt |
| replace global-statistics aggregation with weighted average (ablation) | FedAvg + HBN + Simple-CNN · CIFAR-10, φ=0.6, B=4 | test accuracy (%): 78.22 -> 77.90 | Hurt |
| use only global statistics for normalisation (ablation) | FedAvg + HBN + Simple-CNN · CIFAR-10, φ=0.6, B=4 | test accuracy (%): 78.22 -> 76.87 | Hurt |
| use only batch statistics for normalisation (ablation) | FedAvg + HBN + Simple-CNN · CIFAR-10, φ=0.6, B=4 | test accuracy (%): 78.22 -> 76.33 | Hurt |
| fix global-to-batch statistics ratio at 1:1 (ablation) | FedAvg + HBN + Simple-CNN · CIFAR-10, φ=0.6, B=4 | test accuracy (%): 78.22 -> 77.49 | Hurt |
| use a smaller moving-average momentum when client activation is low (ablation) | HBN · CIFAR-10, φ=0.6, B=32, low client activation | test accuracy | Helped |
Surprising findings
- Standard BN remains competitive in most heterogeneous settings, and FBN collapses on Tiny-ImageNet (4.90% at phi=0.1) with the best learning rate.
- Using the proposed separated update without the moving average is worse, suggesting moving-average global statistics help under low client activation.
Limitations
- Experiments are limited to image classification on CIFAR and Tiny-ImageNet with simulated Dirichlet partitions.
- Global statistics are one round stale, and the method adds a synchronisation round for statistics.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: top-1 test accuracy
Baselines: BN GN LN FedFN FixBN FBN FedProx FedAdam Scaffold MOON FedSAM FedACG
Builds on: FedAvg Batch Normalisation FixBN Federated Batch Normalisation FedTAN
Closest papers by meaning
- Population Normalization for Federated Learning (CVPR 2025 · similarity 0.93)
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization (ICLR 2024 · similarity 0.88)
- Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.86)
- FedUV: Uniformity and Variance for Heterogeneous Federated Learning (CVPR 2024 · similarity 0.86)
- Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Data (NeurIPS 2024 · similarity 0.86)
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.