Research map / Federated learning and privacy / Communication-efficient FL
Federated Optimization with Doubly Regularized Drift Correction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits DANE and proposes DANE+ (inexact local solvers, flexible averaging) and FedRed (doubly regularized drift correction with probabilistic communication), showing communication reduction depending on averaged Hessian dissimilarity.
Problem
Federated/distributed optimization suffers from client drift and communication costs; existing drift-correction methods have communication complexity dependent on smoothness L and do not uniformly beat gradient descent. The paper asks whether communication can be reduced in terms of Hessian similari
Approach
Revisit DANE as regularized drift correction, extend it to DANE+ with inexact local solvers and flexible averaging, and propose FedRed, which adds a second regularizer around a large stepsize eta so local subproblems are well conditioned and communicates only with probability p, enabling one-gradient-step local updates (FedRed-GD) with communication reduced by about L/delta.
Key results
- On synthetic quadratics with L/delta_A ~ L/delta_B >= 20, DANE+-GD and FedRed-GD need roughly 20x fewer communication rounds than GD to reach the same suboptimality.
- FedRed-GD matches GD's total gradient computation complexity K=O(L/eps) while communication is reduced to O(delta_A/eps) in the convex case, a factor L/delta_A better than GD.
- On LIBSVM regularized logistic regression with n=5 Dirichlet-split clients, DANE+-GD and FedRed-GD consistently converge faster than Scaffold, Scaffnew, FedProx and GD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DANE+-GD instead of centralized GD under Hessian similarity | DANE+-GD · Synthetic quadratic problems | communication rounds: approximately 20 times fewer communication rounds | Helped |
| Use FedRed-GD instead of centralized GD under Hessian similarity | FedRed-GD · Synthetic quadratic problems | gradient computation cost: at the same scale as GD | No clear effect |
| Add the additional regularizer to the DANE+ local objective and skip communication probabilistically | FedRed · Non-convex optimization | local computational complexity | Helped |
| Use FedRed-GD rather than DANE+-GD local solving | FedRed-GD · Strongly-convex and general convex optimization | local steps per round: $\frac{L}{\delta_A}$ | Helped |
| Use regularized drift correction with similarity-dependent rates instead of centralized GD | DANE · General convex optimization | communication rounds: $\cO(\frac{\delta_A R_0^2}{\varepsilon})$ vs. $\cO\bigl( \frac{LR_0^2}{\varepsilon} \bigr)$ | Helped |
| Use FedRed-GD instead of GD when $L/\delta_A\approx L/\delta_B\gtrsim 20$ | FedRed-GD · Synthetic strongly-convex, general convex, and non-convex problems | communication rounds: approximately 20 times fewer communication rounds | Helped |
| Set $p=1$ and $\eta=0$ to recover DANE+ from FedRed (ablation) | FedRed · Optimization framework | method equivalence | No clear effect |
| Replace the exact/local subproblem solver with one local gradient step (ablation) | FedRed-GD · Smooth non-convex optimization | gradient computation complexity: $K=\cO(\frac{L}{\varepsilon})$ | No clear effect |
Surprising findings
- DANE+-GD's total local computation is worse than GD by a factor of delta_B/eps in the non-convex case, showing that the communication gain can come at a computational cost.
- Standard averaging is argued to be less effective than randomized averaging for non-convex problems when clients are similar.
Limitations
- Theory for FedRed-GD with stochastic gradients and inexact control variates is not provided; deep learning experiments are deferred and not extensive.
- Client sampling and decentralized settings are left as future work.
- DANE+ non-convex analysis incurs worse local computation than GD.
Experimental setup
Datasets: LIBSVM datasets synthetic quadratic functions
Benchmarks: LIBSVM binary classification (four datasets) synthetic quadratic and non-convex problems
Metrics: communication rounds to target suboptimality total local gradient computations
Baselines: GD Scaffold Scaffnew FedProx DANE+-GD
Builds on: DANE Scaffold FedProx Scaffnew SVRP
Closest papers by meaning
- FedSplit: an algorithmic framework for fast federated optimization (NeurIPS 2020 · similarity 0.91)
- Stabilized Proximal-Point Methods for Federated Optimization (NeurIPS 2024 · similarity 0.91)
- Adaptive Federated Optimization (ICLR 2021 · similarity 0.90)
- Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning (NeurIPS 2022 · similarity 0.90)
- Decentralized Sporadic Federated Learning: A Unified Algorithmic Framework with Convergence Guarantees (ICLR 2025 · similarity 0.89)
- Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updates (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.