Research map / Federated learning and privacy / Communication-efficient FL
The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes three add-on techniques for low-rank communication-efficient FL: decomposing only model updates (MUD), block-wise Kronecker decomposition (BKD), and aggregation-aware decoupled decomposition (AAD), with a convergence analysis and experiments on four small image datasets.
Problem
Low-rank decomposition methods for bidirectional communication-efficient federated learning lose information and accuracy because they decompose full parameters, are limited in rank, and suffer bias when aggregating sub-matrices.
Approach
Freeze the global weights and learn low-rank model updates (MUD); use block-wise Kronecker products to raise the achievable rank at fewer parameters (BKD); and decouple the trainable factors via fixed shared matrices so that aggregating sub-matrices equals recovering then aggregating (AAD).
Key results
- On CIFAR-100 Non-IID-1 at 1/32 compression, MUD+BKD+AAD reaches 41.2% versus 49.8% for FedAvg and 28.4% for FedLMT.
- Adding AAD to MUD consistently improves accuracy over freezing one factor under the same communication cost (e.g., CIFAR-10 Non-IID-1: 73.3 vs 71.1).
- With 8x compression, FedMUD+BKD+AAD is reported to reach accuracy comparable to FedAvg.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add BKD to FedMUD (ablation) | FedMUD · FMNIST, SVHN, CIFAR-10, and CIFAR-100 (Non-IID-1 and Non-IID-2) | accuracy: 87.9($\pm$ 0.2) -> 88.2($\pm$ 0.1); 85.8($\pm$ 0.4) -> 86.7($\pm$ 0.6); 81.3($\pm$ 0.8) -> 84.8($\pm$ 0.5); 78.5($\pm$ 0 | Helped |
| add AAD to FedMUD (ablation) | FedMUD · Non-IID-1: FMNIST, SVHN, CIFAR-10, CIFAR-100 | accuracy: 87.9 -> 88.5; 81.3 -> 84.7; 68.9 -> 73.3; 29.4 -> 37.4 | Helped |
| add BKD and AAD to FedMUD (ablation) | FedMUD · Non-IID-1 and Non-IID-2 across FMNIST, SVHN, CIFAR-10, CIFAR-100 | accuracy: 87.9($\pm$ 0.2) -> 89.0($\pm$ 0.2); 85.8($\pm$ 0.4) -> 87.6($\pm$ 0.2); 81.3($\pm$ 0.8) -> 86.6($\pm$ 0.3); 78.5($\pm$ 0 | Helped |
| use AAD rather than freezing one factor (ablation) | FedMUD · Non-IID-1: FMNIST, SVHN, CIFAR-10, CIFAR-100 | accuracy: 88.5 -> 88.5; 81.0 -> 84.7; 71.1 -> 73.3; 35.3 -> 37.4 | Mixed |
| increase the reset interval above 1 (ablation) | FedMUD · classification datasets; reset-interval experiments | accuracy | Mixed |
| use an inappropriate initialization magnitude (ablation) | FedMUD · classification datasets; initialization ablation | accuracy | Hurt |
| use BKD instead of standard low-rank decomposition (ablation) | FedMUD · classification datasets; initialization ablation | accuracy | Helped |
| decrease compression strength to 8-fold compression (ablation) | FedMUD+BKD+AAD · classification datasets; compression-ratio ablation | accuracy | Helped |
Surprising findings
- The reset interval of 1 is best; a slight accuracy gain is occasionally seen at interval 2 or 4 due to data heterogeneity.
Limitations
- Experiments use small CNNs on small image datasets rather than large models.
- Initialization scale requires per-method tuning; BKD depends on initialization values.
- Theoretical analysis excludes BKD and AAD and relies on several assumptions.
Experimental setup
Datasets: FMNIST SVHN CIFAR-10 CIFAR-100 TinyImageNet
Benchmarks: FMNIST SVHN CIFAR-10 CIFAR-100 TinyImageNet
Metrics: test accuracy convergence speed
Baselines: FedAvg FedHM FedLMT FedPara EF21-P FedBAT
Builds on: FedLMT FedHM FedPara LoRA EF21-P
Closest papers by meaning
- Layer-wise Update Aggregation with Recycling for Communication-Efficient Federated Learning (NeurIPS 2025 · similarity 0.88)
- Achieving Lossless Gradient Sparsification via Mapping to Alternative Space in Federated Learning (ICML 2024 · similarity 0.87)
- Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization (ICLR 2025 · similarity 0.87)
- Stochastic Controlled Averaging for Federated Learning with Communication Compression (ICLR 2024 · similarity 0.87)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.86)
- In-depth Analysis of Low-rank Matrix Factorisation in a Federated Setting (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.