Research map / Federated learning and privacy / Communication-efficient FL
Adaptive Federated Learning with Auto-Tuned Clients
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Δ-SGD, a locally adaptive client step-size rule for federated SGD that adapts each client's step size to its local smoothness without tuning, with a nonconvex convergence analysis and experiments across vision and text FL tasks.
Problem
Client-side step-size tuning in federated learning is hard because local objectives differ across clients (non-iid data, varying sizes), and standard SGD variants and adaptive optimizers need task-specific retuning.
Approach
Each client sets its own step size from the ratio of iterate change to gradient change (a local-smoothness estimate), capped by a growth rule sqrt(1+θ)·η_prev, so step sizes adapt locally and can increase during local iterations; the rule is agnostic to the loss and server optimizer.
Key results
- Δ-SGD achieves 84.5% on CIFAR-10 with ResNet-18 (α=0.1) and 64.1% on CIFAR-100 with ResNet-50 (α=0.1) using step sizes tuned only on CIFAR-10, outperforming all other client optimizers in those settings.
- With FedProx on CIFAR-10 (ResNet-18) at α=0.01, Δ-SGD reaches 33.8% vs. 31.3% for the best baseline (SGD with LR decay).
- On text classification with DistilBERT, Δ-SGD reaches 98.6% on DBpedia while SGDM, Adam, and Adagrad collapse to chance-level accuracy using step sizes tuned on CIFAR-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use client-adaptive Δ-SGD instead of the strongest listed baseline | Federated image classification · CIFAR-10 ResNet-18, α=0.1 | accuracy: 85.0 -> 84.5 | No clear effect |
| Use client-adaptive Δ-SGD instead of the strongest listed baseline | Federated image classification · CIFAR-100 ResNet-50, α=0.1 | accuracy: 58.2 -> 64.1 | Helped |
| Use client-adaptive Δ-SGD instead of the strongest listed baseline | Federated image classification · CIFAR-100 ResNet-18, α=0.01 | accuracy: 30.5 -> 31.8 | Helped |
| Use client-adaptive Δ-SGD instead of the strongest listed baseline | Federated image classification · FMNIST CNN, α=0.1 | accuracy: 86.1 -> 86.4 | Helped |
| Use client-adaptive Δ-SGD instead of the strongest listed baseline | Federated text classification · Agnews DistillBERT, α=1 | accuracy: 91.5 -> 90.7 | Hurt |
| Use client-adaptive Δ-SGD instead of the strongest listed baseline | Federated text classification · DBpedia DistillBERT, α=1 | accuracy: 98.9 -> 98.6 | Hurt |
| Add the FedProx proximal loss to Δ-SGD (ablation) | Δ-SGD · CIFAR-10 ResNet-18, α=0.01 | accuracy | Mixed |
| Add the FedProx proximal loss to Δ-SGD (ablation) | Δ-SGD · CIFAR-100 ResNet-50, α=0.01 | accuracy | Mixed |
| Vary the Δ-SGD δ parameter (ablation) | Δ-SGD · Federated classification | accuracy | No clear effect |
Surprising findings
- Adam, Adagrad, and SGDM with step sizes tuned on CIFAR-10 collapse to chance accuracy on text classification, showing step sizes do not transfer across domains.
- SGD with LR decay outperforms Δ-SGD on CIFAR-10 ResNet-18 at α=1 and α=0.1 (e.g., 87.8 vs 89.8 at α=1 is actually lower for SGD), with Δ-SGD second-best in the MNIST/CIFAR-10 comparison at α=0.1 in some settings.
Limitations
- Convergence analysis relies on a non-standard strong-growth-of-dissimilarity assumption (rho = O(1)) and requires step-size conditions (gamma, eta_0) used only for theory.
- Δ-SGD does not uniformly win; SGD with LR decay or SGDM is best on some settings such as CIFAR-10 at α=0.1 and MNIST.
- Experiments use a fixed hyperparameter tuning on a single task (CIFAR-10, ResNet-18, α=0.1) and a small set of models and datasets.
Experimental setup
Datasets: MNIST FMNIST CIFAR-10 CIFAR-100 DBpedia AGnews
Benchmarks: MNIST FMNIST CIFAR-10 CIFAR-100 DBpedia AGnews
Metrics: test accuracy
Baselines: SGD SGDM Adam Adagrad SPS FedProx FedAdam
Builds on: FedAvg Adam Adagrad Stochastic Polyak step size Malitsky-Mishchenko adaptive gradient descent
Closest papers by meaning
- FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participation (NeurIPS 2026 · similarity 0.90)
- Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable (NeurIPS 2025 · similarity 0.89)
- Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability (NeurIPS 2024 · similarity 0.89)
- FedECADO: A Dynamical System Model of Federated Learning (ICML 2025 · similarity 0.89)
- Principled Federated Domain Adaptation: Gradient Projection and Auto-Weighting (ICLR 2024 · similarity 0.89)
- Communication-Efficient Federated Learning with Accelerated Client Gradient (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.