Research map / Federated learning and privacy / Communication-efficient FL
z-SignFedAvg: A Unified Stochastic Sign-Based Compression for Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes z-SignFedAvg, a federated averaging algorithm with stochastic sign compression using z-distribution noise injected before the sign, unifying prior stochastic sign methods and enabling multiple local SGD steps with improved convergence.
Problem
Federated learning suffers from high communication cost; sign-based gradient compression can diverge under heterogeneous data, existing fixes (error feedback, stochastic sign) converge slowly, and none supports multiple local SGD steps per round like FedAvg.
Approach
Injects symmetric zero-mean noise (z-distribution, with uniform as z→∞ and Gaussian as z=1) before the sign operator so the compressed signal is asymptotically unbiased with controllable bias-variance tradeoff, yielding the first sign-based FedAvg with local updates.
Key results
- z-SignFedAvg achieves convergence rate O((nτ)^(-z/(2z+1))), giving O((nτ)^(-1/3)) for z=1 and O((nτ)^(-1/2)) for z=∞, matching uncompressed FedAvg under uniform noise with sufficiently large σ.
- On non-i.i.d. MNIST, 1-SignSGD and ∞-SignSGD clearly outperform EF-SignSGDwM and Sto-SignSGDwM and perform close to uncompressed SGDwM.
- On EMNIST, 1-SignFedAvg slightly outperforms uncompressed FedAvg, and on CIFAR-10 it dominates FedPAQ and Drive+ across bit budgets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use stochastic sign compression with multiple local SGD updates | z-SignFedAvg · EMNIST | test accuracy | Helped |
| Use stochastic sign compression with multiple local SGD updates | z-SignFedAvg · CIFAR-10 | test accuracy | Helped |
| Use stochastic sign compression | z-SignFedAvg · non-i.i.d. MNIST | test accuracy | Helped |
| Use vanilla SignSGD without noise perturbation | SignSGD · consensus problem | objective value | Hurt |
| Use an input-dependent noise scale | Sto-SignSGD · consensus problem | convergence speed | Hurt |
| Use 1-SignSGD or ∞-SignSGD instead of uncompressed GD | sign-based SGD · consensus problem | convergence speed | Hurt |
| Vary the noise scale (ablation) | 1-SignSGD and ∞-SignSGD · consensus problem | objective value | Mixed |
| Use multiple local SGD steps (ablation) | FedAvg and 1-SignFedAvg · EMNIST | test accuracy | Helped |
| Use multiple local SGD steps (ablation) | FedAvg and 1-SignFedAvg · CIFAR-10 | test accuracy | Helped |
| Use low-precision sign compression instead of unbiased quantization | 1-SignFedAvg · EMNIST and CIFAR-10 | test accuracy | Helped |
| Use 1-SignFedAvg instead of FedPAQ or Drive+ | 1-SignFedAvg · CIFAR-10 | test accuracy | Helped |
| Use Gaussian noise instead of uniform noise (ablation) | z-SignFedAvg · EMNIST and CIFAR-10 | performance | No clear effect |
Surprising findings
- 1-SignFedAvg can even outperform uncompressed FedAvg on EMNIST.
- The best noise scale in experiments is much smaller than the theoretically predicted one, since minibatch gradient noise itself acts as perturbation.
Limitations
- Requires stronger assumptions on higher-order moments of minibatch gradient noise (bounded moments or bounded infinity-norm noise).
- Uniform-noise variant only converges when noise scale exceeds E(G+Q∞); theoretical noise thresholds are hard to compute in practice and σ is tuned as a hyperparameter.
- Experiments are on small models and datasets (2-layer CNN, ResNet18 on CIFAR-10).
Experimental setup
Datasets: MNIST EMNIST CIFAR-10
Benchmarks: MNIST EMNIST CIFAR-10
Metrics: training loss test accuracy test accuracy vs. transmitted bits squared l2 gradient norm
Baselines: SignSGD Sto-SignSGD EF-SignSGDwM SGDwM FedAvg FedPAQ Drive+
Builds on: FedAvg SignSGD Sto-SignSGD error feedback
Closest papers by meaning
- Noisy SIGNSGD Is More Differentially Private Than You (Might) Think (ICML 2025 · similarity 0.91)
- Stochastic Sign Descent Methods: New Algorithms and Better Theory (ICML 2021 · similarity 0.90)
- Stochastic Controlled Averaging for Federated Learning with Communication Compression (ICLR 2024 · similarity 0.88)
- EFSkip: A New Error Feedback with Linear Speedup for Compressed Federated Learning with Arbitrary Data Heterogeneity (AAAI 2025 · similarity 0.87)
- Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction (NeurIPS 2024 · similarity 0.86)
- Don't Compress Gradients in Random Reshuffling: Compress Gradient Differences (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.