Research map / Federated learning and privacy / Communication-efficient FL
Momentum Benefits Non-iid Federated Learning Simply and Provably
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that adding momentum to local SGD steps lets FedAvg converge under unbounded data heterogeneity with constant local learning rate, and that momentum makes SCAFFOLD provably faster under partial participation, with variance-reduced extensions achieving state-of-the-art rates.
Problem
Federated learning with non-iid clients suffers from client drift and slow convergence; existing FedAvg/SCAFFOLD analyses typically require bounded data heterogeneity, and prior improvements often need impractical algorithmic changes.
Approach
Introduce momentum only into local SGD steps, using a global gradient estimate g^r as the anchor (g_i = beta*grad + (1-beta)*g^r) so clients are pulled toward a shared direction, removing the need for bounded heterogeneity; analogous momentum is applied to SCAFFOLD and to variance-reduced (STORM-style) variants.
Key results
- FedAvg-M converges under unbounded data heterogeneity with constant local learning rate, achieving rate (L Delta sigma^2/(NKR))^{1/2} + L Delta/R, matching the lower bound.
- SCAFFOLD-M achieves up to N^{2/9} times fewer communication rounds than SCAFFOLD when S ~ N^{2/3} in the noiseless case.
- Experiments on CIFAR-10 with MLP and ResNet-18 show FedAvg-M and SCAFFOLD-M significantly outperform FedAvg and SCAFFOLD, with larger gains under severe heterogeneity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add momentum to FedAvg local updates | FedAvg · CIFAR-10, three-layer MLP, full client participation | test loss | Helped |
| Add momentum to SCAFFOLD local updates | SCAFFOLD · CIFAR-10, three-layer MLP, partial client participation | test loss | Helped |
| Add momentum to FedAvg local updates | FedAvg · CIFAR-10, ResNet-18, severe data heterogeneity | test accuracy | Helped |
| Add momentum to SCAFFOLD local updates | SCAFFOLD · CIFAR-10, ResNet-18, severe data heterogeneity | test accuracy | Helped |
| Add momentum to FedAvg | FedAvg · Full client participation, non-convex FL | convergence rate | Helped |
| Add momentum to SCAFFOLD | SCAFFOLD · Partial client participation, non-convex FL | convergence rate: N^{2/9} | Helped |
| Add variance-reduced momentum to FedAvg | FedAvg · Full client participation, variance reduction | convergence rate | Helped |
| Add variance-reduced momentum to SCAFFOLD | SCAFFOLD · Partial client participation, variance reduction | convergence rate | Helped |
| Use momentum coefficient β=1, turning off momentum (ablation) | FedAvg-M · Full client participation | convergence rate | Hurt |
| Use momentum coefficient β=1, recovering FedAvg (ablation) | FedAvg-M-VR · Full client participation, variance reduction | convergence rate | Hurt |
| Use momentum coefficient β=1, turning off momentum (ablation) | SCAFFOLD-M · Partial client participation | convergence rate | Hurt |
Surprising findings
- FedAvg with momentum converges without any bounded data heterogeneity assumption even with constant local learning rate, whereas prior FedAvg analyses required bounded heterogeneity even with diminishing rates.
Limitations
- Momentum coefficient and learning rates require tuning by grid search and theoretical settings depend on problem constants (e.g., beta depends on sigma, R).
- Experiments limited to CIFAR-10 with small MLP and ResNet-18 and N=10 clients.
- Variance-reduced variants require sample-wise smoothness assumption beyond standard smoothness.
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10
Metrics: test loss test accuracy communication rounds to convergence
Baselines: FedAvg SCAFFOLD MB-STORM CE-LSGD FedCM MimeLiteMVR STEM LED VRL-SGD
Builds on: FedAvg SCAFFOLD STORM FedCM
Closest papers by meaning
- On the Role of Server Momentum in Federated Learning (AAAI 2024 · similarity 0.91)
- FedWMSAM: Fast and Flat Federated Learning via Weighted Momentum and Sharpness-Aware Minimization (NeurIPS 2025 · similarity 0.90)
- Momentum-Driven Adaptivity: Towards Tuning-Free Asynchronous Federated Learning (ICML 2025 · similarity 0.90)
- Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable (NeurIPS 2025 · similarity 0.90)
- Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg (ICML 2025 · similarity 0.89)
- A New Theoretical Perspective on Data Heterogeneity in Federated Optimization (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.