Research map / Federated learning and privacy / Communication-efficient FL
Achieving Lossless Gradient Sparsification via Mapping to Alternative Space in Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Maps local FL gradients into an SVD-derived orthogonal space built from input activations and output gradients so that Top-k and Sparse-Binary compressors can sparsify more losslessly, with convergence preserved.
Problem
Reducing uplink communication in federated learning, where gradient compression has been restricted to the original parameter space, limiting achievable compression rate.
Approach
Rewrites the layer weight gradient as an outer product of output gradient and input activation, then applies orthogonal transforms (from SVD of their covariance matrices, estimated on unlabeled public data at the server) so the gradient becomes sparser in a new space while remaining information-preserving and invertible.
Key results
- On CIFAR100, the mapping reduces bits to reach 41% accuracy by about 245.6x (LMap) versus FedAvg (23.09 GiB to 0.0940 GiB).
- On CIFAR10, reaching 78% accuracy takes 1.014 GiB under FedAvg versus 0.0068 GiB with LMap (about 149x reduction).
- kSB + Map achieves higher final accuracy than uncompressed FedAvg in some settings, while the convergence rate matches FedAvg's O(1/sqrt(NTH)) with linear speedup in clients.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed mapping with k-Sparse-Binary instead of uncompressed FedAvg | Federated learning on CIFAR10 · CIFAR10, target accuracy 78% | communication bits: 1.014 -> 0.0068 GiB | Helped |
| Use the proposed mapping with k-Sparse-Binary instead of uncompressed FedAvg | Federated learning on CIFAR100 · CIFAR100, target accuracy 41% | communication bits: 23.09 -> 0.0940 GiB | Helped |
| Construct the mapping from a different dataset rather than the target dataset (ablation) | Mapping + k-Sparse-Binary · CIFAR10, target accuracy 78% | communication bits: 0.0068 -> 0.0075 GiB | No clear effect |
| Construct the mapping from a different dataset rather than the target dataset (ablation) | Mapping + k-Sparse-Binary · CIFAR100, target accuracy 41% | communication bits: 0.0940 -> 0.0899 GiB | No clear effect |
| Vary the public-data ratio from 0.1% to 5% to construct the mapping (ablation) | Mapping + k-Sparse-Binary · CIFAR100, α = 0.6, target accuracy 41% | communication bits: 0.0956 (0.1%); 0.0903 (0.5%); 0.0940 (1%); 0.0944 (5%) | No clear effect |
Surprising findings
- Sparse-Binary compression combined with the mapping can exceed the accuracy of full-gradient FedAvg, which does not hold for Sparse-Binary in the original space.
- The mapping built from only 0.1% public data still yields effective compression on CIFAR100.
Limitations
- Requires unlabeled public data at the server to construct the mapping, which conflicts with strict privacy settings.
- Adds server-side computation and downlink communication for broadcasting the mapping, and the mapping must be periodically reconstructed.
Experimental setup
Datasets: SVHN CIFAR10 CIFAR100 TinyImageNet
Benchmarks: SVHN CIFAR10 CIFAR100 TinyImageNet
Metrics: test accuracy number of bits uploaded to reach target accuracy
Baselines: Top-k k-Sparse-Binary FedDQ FedPara FedAvg PD1 PD2
Builds on: Top-k sparsification sparse binary compression (SBC) error feedback FedAvg
Closest papers by meaning
- SpaFL: Communication-Efficient Federated Learning With Sparse Models And Low Computational Overhead (NeurIPS 2024 · similarity 0.88)
- EFSkip: A New Error Feedback with Linear Speedup for Compressed Federated Learning with Arbitrary Data Heterogeneity (AAAI 2025 · similarity 0.88)
- SparsyFed: Sparse Adaptive Federated Learning (ICLR 2025 · similarity 0.88)
- The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning (ICML 2025 · similarity 0.87)
- LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression (ICLR 2025 · similarity 0.87)
- Stochastic Controlled Averaging for Federated Learning with Communication Compression (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.