Research map / Federated learning and privacy / Communication-efficient FL
SparsyFed: Sparse Adaptive Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SparsyFed, a federated sparse training method combining layer-wise activation pruning, Powerpropagation weight re-parameterization, and a single-parameter global Top-K pruning step to reach up to 95% sparsity with dynamic masks under heterogeneous data.
Problem
Sparse training in cross-device federated learning is rarely used because data heterogeneity makes clients hard to agree on sparse masks, fixed mask methods lack adaptivity, and extra hyperparameters are hard to tune in FL.
Approach
Train sparse models on clients via layer-wise activation pruning in the backward pass, induce sparsity with Powerpropagation weight re-parameterization so that masks converge across clients while remaining dynamic, and apply a global Top-K pruning of pseudo-gradients to meet a target sparsity with only one hyperparameter.
Key results
- SparsyFed at 95% sparsity on CIFAR-10 with LDA alpha=0.1 reaches 75.95% vs 61.55% for ZeroFL and 72.08% for FLASH, and at alpha=1.0 reaches 82.60% vs 79.36% for FLASH (dense 83.70%).
- Reports up to 19.29x lower communication cost than the dense model and 1.66x lower than ZeroFL on CIFAR-100 at 95% sparsity with alpha=0.1.
- At 99.9% sparsity on CIFAR-10 (alpha=1.0), SparsyFed reaches 51.39% vs 36.07% for FLASH and 17.66% for ZeroFL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SparsyFed instead of the strongest sparse baseline | ResNet-18 · CIFAR-10, sparsity 0.95, α=1.0 and α=0.1 | accuracy: 74.71 ± 3.29 / 60.00 ± 4.66 -> 82.60 ± 1.58 / 75.95 ± 3.39 | Helped |
| Use SparsyFed instead of the strongest sparse baseline | ResNet-18 · CIFAR-100, sparsity 0.95, α=1.0 and α=0.1 | accuracy: 52.81 ± 1.72 / 48.27 ± 2.70 | Helped |
| Use SparsyFed instead of the strongest sparse baseline | ResNet-18 · Speech Commands, sparsity 0.95, α=1.0 and α=0.1 | accuracy: 89.14 ± 1.15 / 75.46 ± 2.24 | Helped |
| Add activation pruning (ablation) | SparsyFed · CIFAR-10, sparsity 0.95 across α=10^3, 1.0, and 0.1 | accuracy: 83.80 ± 0.90 -> 84.25 ± 1.38; 81.53 ± 2.10 -> 82.6 ± 1.58; 75.29 ± 2.64 -> 75.95 ± 3.39 | Mixed |
| Add activation pruning (ablation) | SparsyFed · CIFAR-10, sparsity 0.999 across α=10^3, 1.0, and 0.1 | accuracy: 62.12 ± 1.74 -> 55.24 ± 2.09; 62.67 ± 2.19 -> 51.39 ± 3.191; 49.43 ± 2.45 -> 43.68 ± 7.61 | Hurt |
| Use Powerpropagation instead of fixed-mask training, spectral re-parameterization, or no re-parameterization (ablation) | SparsyFed with sparse activations · CIFAR-10, LDA α=1 | accuracy | Helped |
| Use a dynamically computed Powerpropagation exponent instead of a fixed exponent (ablation) | SparsyFed · Dynamic sparse training | accuracy | Mixed |
| Use SparsyFed instead of naive federated Powerpropagation with pruning only after training (ablation) | ResNet-18 · CIFAR-10, sparsity 0.900–0.999 across α=10^3, 1.0, and 0.1 | accuracy: 71.41 ± 7.01 -> 84.31 ± 0.86; 66.70 ± 3.76 -> 82.13 ± 1.53; 45.82 ± 6.32 -> 75.00 ± 2.78 (0.900) | Helped |
Surprising findings
- At 90-95% sparsity SparsyFed sometimes outperforms the dense model, e.g. 82.60% vs 83.70% dense is close but Speech Commands at 90% reaches 92.32% vs 91.49% dense.
- Naive federated Powerpropagation (pruning only after training) collapses to near-random accuracy at 95% sparsity (e.g. 35.02% on CIFAR-10 IID).
Limitations
- Experiments limited to ResNet-18 on CIFAR-10/100 and Speech Commands with 100 simulated clients and 10 sampled per round.
- Accuracy drops noticeably at 99% sparsity and collapses at 99.9%, and activation pruning causes degradation at extreme sparsity.
- Hyperparameter-free Powerpropagation exponent variant still underperforms the best fixed beta.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Speech Commands
Benchmarks: CIFAR-10 CIFAR-100 Speech Commands
Metrics: test accuracy communication cost (non-zero parameters exchanged) IoU of global masks across rounds global model sparsity
Baselines: Top-K ZeroFL FLASH Naive Powerpropagation dense ResNet-18
Builds on: Powerpropagation SWAT ZeroFL FLASH Sparse Ternary Compression
Closest papers by meaning
- SpaFL: Communication-Efficient Federated Learning With Sparse Models And Low Computational Overhead (NeurIPS 2024 · similarity 0.91)
- An Efficient and Accurate Dynamic Sparse Training Framework Based on Parameter-Freezing (AAAI 2025 · similarity 0.91)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.89)
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.88)
- Revisiting Ensembling in One-Shot Federated Learning (NeurIPS 2024 · similarity 0.88)
- TurboSVM-FL: Boosting Federated Learning through SVM Aggregation for Lazy Clients (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.