Research map / Federated learning and privacy / Communication-efficient FL
On the Role of Server Momentum in Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedGM, a general server-momentum framework for federated learning with stagewise ("constant and drop") hyperparameter scheduling and asynchronous heterogeneous local computing, with convergence analysis and CIFAR-10 experiments.
Problem
FedAvg converges unstably under client drift from data heterogeneity and system heterogeneity; existing server momentum work only covers heavy-ball momentum, uses unrealistic hyperparameter schedules, and assumes homogeneous, synchronous clients.
Approach
A general server momentum formulation (FedGM) with learning rate, momentum factor and instant discount factor that subsumes FedAvg, FedAvgM, FedNAG and other momentum schemes; extended to multistage ("constant and drop") schedules and to an autonomous asynchronous variant where clients run variable local steps and participate at will.
Key results
- On ResNet/CIFAR-10 with grid-searched hyperparameters, FedGM outperforms FedAvgM and FedAvg on both training and testing, while FedAvgM is only marginally better than FedAvg on testing.
- Multistage FedGM with (eta1=2.0, eta2=1.0, eta3=0.5) outperforms single-stage FedGM at any constant eta in training and testing on ResNet/CIFAR-10.
- Autonomous FedGM outperforms Autonomous FedAvg under system heterogeneity and lags the ideal synchronous FedGM only by a small margin.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace FedAvg server updates with general server momentum | FedAvg on ResNet · CIFAR-10 | test performance | Helped |
| replace FedAvg server updates with general server momentum | FedAvg on ResNet · CIFAR-10 | training convergence | Helped |
| use FedAvgM (SHB) instead of FedAvg | FedAvg on ResNet · CIFAR-10 | training convergence | Helped |
| use FedAvgM (SHB) instead of FedAvg | FedAvg on ResNet · CIFAR-10 | test performance | Helped |
| add a stagewise hyperparameter schedule | FedGM · CIFAR-10, ResNet | test performance | Helped |
| add a stagewise hyperparameter schedule | FedGM · CIFAR-10, ResNet | training convergence | Helped |
| enable autonomous participation and heterogeneous asynchronous local computing | FedGM · CIFAR-10, ResNet with system heterogeneity | performance | Helped |
| enable autonomous participation and heterogeneous asynchronous local computing | FedGM · CIFAR-10, ResNet with system heterogeneity | performance | Hurt |
Surprising findings
- FedAvgM converges faster than FedAvg but is only marginally better in testing accuracy.
Limitations
- Non-vanishing sigma_g^2 term without uniform client sampling assumption.
- Impact of local steps K on convergence is not settled by the analysis.
- Experiments limited to ResNet and VGG on CIFAR-10.
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10
Metrics: training loss testing accuracy
Baselines: FedAvg FedAvgM Autonomous FedAvg
Builds on: FedAvg FedAvgM FedOPT quasi-hyperbolic momentum (QHM) SCAFFOLD
Closest papers by meaning
- Momentum Benefits Non-iid Federated Learning Simply and Provably (ICLR 2024 · similarity 0.91)
- Momentum-Driven Adaptivity: Towards Tuning-Free Asynchronous Federated Learning (ICML 2025 · similarity 0.90)
- FedWMSAM: Fast and Flat Federated Learning via Weighted Momentum and Sharpness-Aware Minimization (NeurIPS 2025 · similarity 0.89)
- Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable (NeurIPS 2025 · similarity 0.88)
- Efficient Asynchronous Federated Learning with Prospective Momentum Aggregation and Fine-Grained Correction (AAAI 2024 · similarity 0.87)
- Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.