Research map / Federated learning and privacy / Personalized and heterogeneous FL
FedWMSAM: Fast and Flat Federated Learning via Weighted Momentum and Sharpness-Aware Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedWMSAM, a federated learning method that combines personalized server-aggregated momentum with a single-backprop SAM perturbation guided by global momentum, and a cosine-similarity adaptive weight that shifts from momentum early to SAM late.
Problem
Federated learning under non-IID data must converge quickly under tight communication budgets while generalizing; naively combining momentum and SAM suffers from local-global curvature misalignment and late-stage momentum oscillation.
Approach
Builds a global perturbation direction from server-aggregated personalized momentum (with a SCAFFOLD-style correction) so that local SAM needs only one backward pass, and couples momentum and SAM through a cosine-similarity adaptive factor producing an early-momentum, late-SAM schedule.
Key results
- On CIFAR-10 with Dirichlet beta=0.1, FedWMSAM reaches 76.64% vs 70.05% for FedAvg (+6.59 points).
- On CIFAR-100 with Dirichlet beta=0.1, FedWMSAM reaches 46.46%, improving over FedAvg by +8.31 points.
- FedWMSAM reaches 0.76 accuracy on CIFAR-10 in 241 rounds, versus 426 for FedCM and never for FedAvg/FedSAM; per-round client time is 15.03s versus 26.9-29.8s for double-backprop SAM methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedWMSAM instead of FedAvg | CIFAR-10 and CIFAR-100 under strong non-IID splits · CIFAR-10 Dirichlet β=0.1; CIFAR-10 pathological γ=3; CIFAR-100 Dirichlet β=0.1; CIFAR-100 pathological γ=10 | accuracy: +6.59; +10.2; +8.31; +7.52 points | Helped |
| Add personalized momentum (ablation) | FedWMSAM · CIFAR-10 Dirichlet β=0.1 | accuracy: 0.7265 -> 0.7664 | Helped |
| Add momentum-guided SAM (ablation) | FedWMSAM · CIFAR-10 Dirichlet β=0.1 | accuracy: 0.7326 -> 0.7664 | Helped |
| Add cosine-adaptive weight (ablation) | FedWMSAM with momentum and SAM · CIFAR-10 Dirichlet β=0.1 | accuracy: 0.7556 -> 0.7664 | Helped |
| Add cosine-adaptive weight (ablation) | momentum-only variant of FedWMSAM · CIFAR-10 Dirichlet β=0.1 | accuracy: 0.7478 -> 0.7326 | Hurt |
| Vary perturbation magnitude ρ from 0.01 to larger values (ablation) | FedWMSAM · CIFAR-10 Dirichlet β=0.1 | accuracy: 0.7664 -> 0.7563 -> 0.7244 -> 0.5905 | Hurt |
| Vary perturbation magnitude ρ from 0.01 to larger values (ablation) | MoFedSAM · CIFAR-10 Dirichlet β=0.1 | accuracy: 0.7102 -> 0.5562 -> 0.1000 | Hurt |
| Increase local epochs from 5 to 10 (ablation) | FedWMSAM · CIFAR-10 Dirichlet β=0.1 | accuracy: 0.7664 -> 0.7662 | No clear effect |
Surprising findings
- MoFedSAM collapses at large perturbation radius (accuracy 0.1000 at rho=0.5), while FedWMSAM degrades more gracefully.
- FedGAMMA accuracy on CIFAR-100 pathological gamma=10 setting is very low (0.0198 in the gamma=3 column), indicating instability of some baselines.
Limitations
- Gains are concentrated in strong non-IID settings; differences on easier splits are small.
- Convergence bound relies on standard smoothness and bounded-variance assumptions, and the paper claims it is comparable or sharper than prior bounds without a full tight comparison.
Experimental setup
Datasets: Fashion-MNIST CIFAR-10 CIFAR-100 OfficeHome
Benchmarks: Fashion-MNIST CIFAR-10 CIFAR-100 OfficeHome
Metrics: test accuracy rounds to target accuracy client computation time
Baselines: FedAvg FedCM SCAFFOLD FedSAM MoFedSAM FedGAMMA FedSMOO FedLESAM
Builds on: SAM FedCM SCAFFOLD FedLESAM MoFedSAM
Closest papers by meaning
- Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization (ICML 2024 · similarity 0.91)
- Momentum Benefits Non-iid Federated Learning Simply and Provably (ICLR 2024 · similarity 0.90)
- FedVSSAM: Mitigating Flatness Incompatibility in Sharpness-Aware Federated Learning (NeurIPS 2026 · similarity 0.90)
- Momentum-Driven Adaptivity: Towards Tuning-Free Asynchronous Federated Learning (ICML 2025 · similarity 0.90)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.89)
- Rethinking the Flat Minima Searching in Federated Learning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.