Research map / Federated learning and privacy / Communication-efficient FL
Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeComFL, a federated learning method that uses zeroth-order (SPSA-style) gradient estimates so clients and server exchange only seeds and scalar gradient estimates, making per-round communication dimension-free, with convergence analysis under a low effective-rank assumption.
Problem
Federated learning communication cost scales linearly with model dimension d per round, which is prohibitive for large models such as LLMs over low-bandwidth links.
Approach
Replace first-order updates with zeroth-order SPSA gradient scalars and shared random seeds so that the server and clients reconstruct models from scalars; clients revert local models after local updates and rebuild the global model from the lagged history of seeds and scalars, giving O(1) per-round uplink and downlink communication.
Key results
- On OPT-1.3B full-parameter fine-tuning, DeComFL (P=10) reaches 90.78% on SST-2 with 0.24 MB communicated per client versus 90.38% for FedAvg with 1.27 TB.
- DeComFL matches or exceeds MeZO on most SuperGLUE fine-tuning tasks for OPT-125M and OPT-1.3B, e.g. RTE 57.05% vs 52.91% MeZO on OPT-125M.
- Theory gives O(sqrt(d/mPKR)) standard rate and O(sqrt(kappa/mPR)) dimension-free rate under low effective rank with K=1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DeComFL instead of FedZO (P=10) | OPT-125M fine-tuning · SST-2 | accuracy: 84.11% -> 85.08% | Helped |
| use DeComFL instead of FedZO (P=10) | OPT-125M fine-tuning · CB | accuracy: 74.41% -> 75.00% | Helped |
| use DeComFL instead of FedZO (P=10) | OPT-125M fine-tuning · WSC | accuracy: 59.47% -> 59.59% | No clear effect |
| use DeComFL instead of FedZO (P=10) | OPT-125M fine-tuning · WIC | accuracy: 53.37% -> 53.38% | No clear effect |
| use DeComFL instead of FedZO (P=10) | OPT-125M fine-tuning · RTE | accuracy: 54.16% -> 57.05% | Helped |
| use DeComFL instead of FedZO (P=10) | OPT-125M fine-tuning · BoolQ | accuracy: 61.25% -> 61.60% | Helped |
| use DeComFL instead of FedZO (P=10) | OPT-1.3B fine-tuning · SST-2 | accuracy: 90.33% -> 90.78% | Helped |
| use DeComFL instead of FedZO (P=10) | OPT-1.3B fine-tuning · CB | accuracy: 74.49% -> 75.71% | Helped |
| use DeComFL instead of FedZO (P=10) | OPT-1.3B fine-tuning · WSC | accuracy: 61.11% -> 64.16% | Helped |
| use DeComFL instead of FedZO (P=10) | OPT-1.3B fine-tuning · WIC | accuracy: 56.08% -> 56.14% | No clear effect |
| use DeComFL instead of FedZO (P=10) | OPT-1.3B fine-tuning · RTE | accuracy: 59.21% -> 60.89% | Helped |
| use DeComFL instead of FedZO (P=10) | OPT-1.3B fine-tuning · BoolQ | accuracy: 62.14% -> 62.50% | Helped |
Surprising findings
- Gradient-projection variant with first-order gradients performs worse and is less stable than the zeroth-order DeComFL under the same communication cost.
- Combining LoRA with zeroth-order fine-tuning does not substantially degrade accuracy relative to full-parameter tuning.
Limitations
- Zeroth-order convergence is slower, so total computation is high and more rounds are needed.
- Does not address data heterogeneity beyond standard assumptions; the theory with low effective rank is restricted to K=1.
- Effective rank kappa is hard to estimate and validated only indirectly for LLMs.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10 SST-2 CB WSC WIC RTE BoolQ
Benchmarks: MNIST Fashion-MNIST SST-2 CB WSC WIC RTE BoolQ
Metrics: test accuracy communication cost (bytes per client) training loss vs rounds
Baselines: FedAvg FedZO MeZO FedCom FedAvg+Top-k
Builds on: FedAvg FedZO MeZO SPSA
Closest papers by meaning
- Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.89)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.89)
- SpaFL: Communication-Efficient Federated Learning With Sparse Models And Low Computational Overhead (NeurIPS 2024 · similarity 0.88)
- Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimation (AAAI 2025 · similarity 0.88)
- FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.87)
- An Efficient and Accurate Dynamic Sparse Training Framework Based on Parameter-Freezing (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.