Research map / Federated learning and privacy / Communication-efficient FL
Efficient Adaptive Federated Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedAda2 and FedAda2+ — jointly adaptive federated optimizers that avoid transmitting server preconditioners by zero-initializing client preconditioners, with FedAda2+ adding SM3 memory-efficient client optimizers, and proves non-convex convergence at O(T^-1/2).
Problem
Joint server- and client-side adaptive federated optimization is effective but incurs high communication (transmitting preconditioners) and client memory costs, which hinders cross-device deployment.
Approach
Zero-initialize each client's local preconditioners every round instead of transmitting server preconditioners (FedAda2), and further compress client-side second-moment statistics with SM3 (FedAda2+), while retaining convergence guarantees for non-convex objectives under full-batch gradients and partial participation.
Key results
- FedAda2 matches costly joint adaptivity on CIFAR-100 and FEMNIST while transmitting fewer bits; joint adaptivity outperforms server-only adaptive baselines across datasets.
- FedAda2+ with SM3 needs only 0.48% additional client memory for second-moment statistics on ViT, a 99% reduction versus the 1x overhead of standard client adaptivity.
- Theory shows FedAda2 and FedAda2+ converge at O(T^-1/2) for non-convex L-smooth objectives under full-batch client gradients and partial participation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add client-side adaptivity to FedAvg | Federated training · CIFAR-100, GLD-23K, and FEMNIST | test accuracy | Helped |
| Add joint server- and client-side adaptivity | FedAvg and server-only adaptive baselines · CIFAR-100, GLD-23K, and FEMNIST | test accuracy | Helped |
| Stop transmitting server preconditioners and initialize client preconditioners from zero (ablation) | Costly Joint Adaptivity · CIFAR-100, GLD-23K, and FEMNIST | test accuracy | No clear effect |
| Use SM3-compressed client preconditioners (ablation) | FedAda² · ViT client-side second-moment storage | additional memory: 1× -> 0.48\% | Helped |
| Use SM3 memory-efficient client optimizers (ablation) | FedAda² · CIFAR-100, GLD-23K, and FEMNIST | test accuracy | No clear effect |
| Increase the number of local epochs (ablation) | FedAda²+ · Vision model transfer learning | relative performance versus FedAdam | Hurt |
| Transmit server-side Adam preconditioners to client-side AdaGrad (ablation) | Costly Joint Adaptivity · Asymmetric server-client adaptivity | test performance | No clear effect |
| Use delayed local preconditioner updates (ablation) | FedAda² and FedAda²+ · Image and text training | training performance | No clear effect |
| Remove server-preconditioner transmission (ablation) | FedAda² · StackOverflow hyperparameter sweeps | test accuracy | Mixed |
| Add client-side adaptivity via AdaGrad | FedAvg · Heavy-tailed client gradient noise | expected global gradient norm squared: ∞ -> \mathcal{O}\left(\frac{1}{T}\right) | Helped |
Surprising findings
- Transmitting an unbalanced server preconditioner (e.g., server Adam to client AdaGrad) does not significantly hurt costly joint adaptivity performance.
- Removing server preconditioner transmission in FedAda2 destabilizes accuracy across hyperparameter sweeps on StackOverflow, while FedAda2+ with SM3 is robust.
Limitations
- Convergence analysis is for deterministic (full-batch client gradient) setting; stochastic guarantees are left to future work.
- FedAda2 without preconditioner transmission can destabilize accuracy under some hyperparameters on StackOverflow.
Experimental setup
Datasets: StackOverflow CIFAR-100 FEMNIST GLD-23K ImageNet-21K
Benchmarks: StackOverflow CIFAR-100 FEMNIST GLD-23K
Metrics: test accuracy test loss communicated bits convergence rate
Baselines: FedAvg FedAdaGrad FedAdam Costly Joint Adaptivity MIME MIMELite
Builds on: FedAdam FedAdaGrad SM3 FedAvg Local Adaptive FedOPT
Closest papers by meaning
- FADAS: Towards Federated Adaptive Asynchronous Optimization (ICML 2024 · similarity 0.89)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.89)
- FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participation (NeurIPS 2026 · similarity 0.89)
- Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability (NeurIPS 2024 · similarity 0.89)
- Communication-Efficient Federated Learning with Accelerated Client Gradient (CVPR 2024 · similarity 0.89)
- Adaptive Federated Learning with Auto-Tuned Clients (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.