Research map / Federated learning and privacy / Communication-efficient FL
FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedHyper, a hypergradient-based scheduler that adapts global, server-side local, and client-side local learning rates in federated learning, with a convergence analysis built on FedNova and experiments on FMNIST, CIFAR10, and Shakespeare.
Problem
Learning rate scheduling in federated learning is hard because both global and local learning rates affect convergence, their optimal values depend on data heterogeneity and training stage, and the performance is sensitive to the initial learning rate.
Approach
Applies hypergradient descent to FL by approximating the hypergradient of the learning rate with inner products of pseudo-global model updates (for the global and server-side rates) and of consecutive local gradients plus a global-update term (for the client-side rate), with clipping bounds for stability.
Key results
- FedHyper-G improves accuracy over the best baseline by 0.45% (FMNIST), 0.58% (CIFAR10), and 0.66% (Shakespeare).
- FedHyper-CL improves over FedAvg with SGD and Adam by 0.44% on FMNIST and 1.18% on CIFAR10, and over FedAvg with local SGD by 1.33% on Shakespeare.
- Under suboptimal initial learning rates, FedHyper raises FedAvg accuracy on Shakespeare from 30% to 45.77% (about 15 points) with initial global rate 0.5 and local rate 0.001.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FedHyper scheduling | FedAvg · FMNIST | accuracy: +0.45% | Helped |
| Add FedHyper scheduling | FedAvg · CIFAR10 | accuracy: +0.58% | Helped |
| Add FedHyper scheduling | FedAvg · Shakespeare | accuracy: +0.66% | Helped |
| Add client-side local learning-rate scheduling | FedAvg with local SGD · FMNIST | accuracy: +0.44% | Helped |
| Add client-side local learning-rate scheduling | FedAvg with local SGD · CIFAR10 | accuracy: +1.18% | Helped |
| Add client-side local learning-rate scheduling | FedAvg with local SGD · Shakespeare | accuracy: +1.33% | Helped |
| Add client-side local learning-rate scheduling | FedAvg with local Adam · Shakespeare | accuracy | No clear effect |
| Add client-side local learning-rate scheduling | FedAvg · Shakespeare, initial global rate 0.5 and local rate 0.001 | accuracy: 30% -> 45.77% | Helped |
| Add the global scheduler to the client-side local scheduler (ablation) | FedHyper-CL · CIFAR10 and Shakespeare | performance | Helped |
| Use the client-side local scheduler instead of the global scheduler (ablation) | FedHyper · Shakespeare | accuracy: 51.08% -> 52.08% | Helped |
| Use the client-side local scheduler instead of the server-side local scheduler (ablation) | FedHyper · Shakespeare | accuracy: 51.70% -> 52.08% | Helped |
| Add FedHyper scheduling | FedAvg · CIFAR10 and Shakespeare, varied initial learning rates | accuracy: 1% - 5% | Helped |
Surprising findings
- The paper reports that FedHyper-G plus FedHyper-CL superposes benefits, but FedHyper-G combined with FedHyper-SL is not shown because both use the same hypergradient, so the effect is not obvious.
Limitations
- Convergence analysis relies on assumptions borrowed from FedNova and on clipping bounds gamma_alpha and gamma_beta set empirically.
- FedHyper-CL adds computational overhead on clients compared with the other schedulers.
- Gains on several benchmarks are small (under 1% in many settings), and results come from a limited set of three datasets and small models.
Experimental setup
Datasets: FMNIST CIFAR10 Shakespeare
Benchmarks: FMNIST CIFAR10 Shakespeare
Metrics: test accuracy convergence speed (rounds)
Baselines: FedAvg FedAdagrad FedAdam FedExp global learning rate decay local learning rate step decay FedAvg with local Adam
Builds on: hypergradient descent (HD) FedAvg FedNova FedExp FedAdam
Closest papers by meaning
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.87)
- Adaptive Federated Learning with Auto-Tuned Clients (ICLR 2024 · similarity 0.87)
- Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable (NeurIPS 2025 · similarity 0.87)
- FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participation (NeurIPS 2026 · similarity 0.87)
- Efficient Adaptive Federated Optimization (NeurIPS 2025 · similarity 0.87)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.