Research map / Federated learning and privacy / Communication-efficient FL
Methods with Local Steps and Random Reshuffling for Generally Smooth Non-Convex Federated Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes three federated methods (Clip-LocalGDJ, CLERR, Clipped RR-CLI) combining local steps, random reshuffling, partial participation and server-side (smoothed) clipping, with convergence guarantees under (L0,L1)-smoothness and the Polyak-Łojasiewicz condition.
Problem
Federated/distributed non-convex optimization under generalized (L0,L1)-smoothness with local steps, random reshuffling and partial participation, without restrictive assumptions such as data homogeneity, bounded variance or cosine relatedness.
Approach
Combine client inner steps (local GD or RR epochs) with small client stepsizes and a server-side clipped GD "jump" step whose stepsize is 1/(c0 + c1||g||), so clipping is applied only to the aggregated pseudogradient once per epoch or meta-epoch rather than at every step.
Key results
- Clipped methods (CLERR, CSO) significantly outperform non-clipped SO and Nastya on the synthetic (L0,L1)-smooth problem, with CLERR better than CSO.
- On ResNet-18/CIFAR-10, jumping and outer clipping showed no clear benefit; CSO performed best, and the heuristic CLERR-h was proposed.
- Experiments show local steps give no positive effect on the synthetic problem, and server-side clipping (C-LGDJ) beats CELGC and CE-FedAvg.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply clipping to the final pseudogradient once per epoch instead of clipping at every step (ablation) | CLERR versus CSO · Synthetic 1D function | function residual | Helped |
| use clipping rather than no clipping | Shuffle Once methods · Synthetic 1D function | function residual | Helped |
| add jumping at the end of each epoch (ablation) | CLERR and Nastya · ResNet-18 on CIFAR-10 | accuracy | No clear effect |
| add clipping on the outer step (ablation) | CLERR · ResNet-18 on CIFAR-10 | accuracy | No clear effect |
| clip at every inner step | CSO · ResNet-18 on CIFAR-10 | accuracy | Helped |
| use CSO's clipping level and inner stepsize, with a tuned outer stepsize (ablation) | CLERR · ResNet-18 on CIFAR-10 | accuracy | No clear effect |
| increase the number of client local steps (ablation) | C-LGDJ and local-step baselines · Synthetic 100D function | function residual | No clear effect |
| clip the aggregated gradient on the server rather than clipping on the client (ablation) | local-step federated methods · Synthetic 100D function | function residual | Helped |
| use client and data random reshuffling and clip only at the end of the meta-epoch | CRR-CLI versus CE-FedAvg-PP · Synthetic 100D function, partial participation | function residual | Helped |
Surprising findings
- Local steps do not help on the synthetic problem; performance worsens or does not improve with more client steps.
- On ResNet-18/CIFAR-10, the proposed CLERR and jumping offer no gain over clipped SO, which performs best.
Limitations
- Experiments are mostly on a small synthetic problem; neural network evaluation is limited to ResNet-18 on CIFAR-10 where the proposed methods show no clear advantage.
- Theory relies on the smoothed/clipped step with inner stepsizes that must be small, and a mistake in the proof affected the choice of inner stepsizes.
- Heterogeneity-related client drift is not addressed; analysis excludes communication compression and incomplete local epochs.
Experimental setup
Datasets: CIFAR-10 synthetic shifted fourth-order functions
Benchmarks: synthetic shifted fourth-order function CIFAR-10 (ResNet-18)
Metrics: function residual test loss gradient norm accuracy
Baselines: SO (shuffle once SGD) Nastya Clipped SO (CSO) CELGC CE-FedAvg CE-FedAvg-PP ResNet-18 SGD-type baselines
Builds on: LocalGD RR-CLI FedAvg Clipped GD under (L0,L1)-smoothness
Closest papers by meaning
- Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data (NeurIPS 2024 · similarity 0.88)
- Federated Optimization with Doubly Regularized Drift Correction (ICML 2024 · similarity 0.88)
- A New Theoretical Perspective on Data Heterogeneity in Federated Optimization (ICML 2024 · similarity 0.88)
- Decentralized Sporadic Federated Learning: A Unified Algorithmic Framework with Convergence Guarantees (ICLR 2025 · similarity 0.88)
- Stabilized Proximal-Point Methods for Federated Optimization (NeurIPS 2024 · similarity 0.87)
- Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.