Research map / Federated learning and privacy / Communication-efficient FL
Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that FedAvg's data-heterogeneity-induced model divergence shrinks as O(n^{-1/2}) with network width, and that infinite-width FedAvg is linear and matches centralized GD under matched iterations.
Problem
Understanding how data heterogeneity (non-IID clients) affects FedAvg convergence and generalization in overparameterized neural networks, and whether increasing width mitigates it.
Approach
Uses NTK-based analysis of multi-layer overparameterized FedAvg with GD (gradient flow approximation) to show model divergence is bounded by O(n^{-1/2}) in width n, so that as width goes to infinity global and local models become linear, have constant NTKs, and FedAvg generalizes identically to centralized GD.
Key results
- Non-IID test accuracy drop on FNN1/FNN2/FNN4/FNN16 is 17.4%, 9.5%, 6.3%, 2.0% vs IID, showing heterogeneity impact shrinks with width.
- Theory bounds model divergence by zeta = O(n^{-1/2}) with probability at least 1-delta_0 over random initialization.
- Experiments show FedAvg and centralized GD outputs are nearly identical under the same number of GD iterations on mini-CIFAR-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| widen the FNN (FNN1, FNN2, FNN4, FNN16) | FedAvg · MNIST | test accuracy under non-IID data relative to IID: 17.4%, 9.5%, 6.3%, and 2.0% drops | Helped |
| widen the CNN (CNN1, CNN2, CNN8, CNN32) | FedAvg · CIFAR-10 | test accuracy under non-IID data relative to IID: 44.9%, 26.7%, 5.1%, and 2.4% drops | Helped |
| widen the ResNet (ResNet1, ResNet2, ResNet4, ResNet16) | FedAvg · CIFAR-10 | test accuracy under non-IID data relative to IID: 44.6%, 29.1%, 18.7%, and 14.8% drops | Helped |
| use FNN32 instead of FNN1 with learning rate scaled as η = η₀/n | FedAvg · MNIST | test accuracy gap between IID and non-IID data | Helped |
| use CNN32 instead of CNN1 with learning rate scaled as η = η₀/n | FedAvg · CIFAR-10 | test accuracy gap between IID and non-IID data | Helped |
Limitations
- Theory relies on infinite-width limit, gradient flow approximation of GD, and small learning rate eta = eta_0/n, which deviate from practical training.
- Main theoretical equivalence and linearization hold only in the infinite-width regime; experiments on large-scale practical settings use cross-entropy and SGD beyond the theory.
- Experiments are limited to MNIST and CIFAR-10 with M=10 clients and binary mini-datasets.
Experimental setup
Datasets: MNIST CIFAR-10 mini-MNIST mini-CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: test accuracy training loss testing loss output difference between FedAvg and linear model
Baselines: FedBN SCAFFOLD FedNova FedMA MOON Fed-ensemble
Builds on: neural tangent kernel FedAvg lazy training
Closest papers by meaning
- A New Theoretical Perspective on Data Heterogeneity in Federated Optimization (ICML 2024 · similarity 0.91)
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.90)
- Momentum Benefits Non-iid Federated Learning Simply and Provably (ICLR 2024 · similarity 0.89)
- Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable (NeurIPS 2025 · similarity 0.89)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.88)
- Rethinking the Flat Minima Searching in Federated Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.