Research map / Federated learning and privacy / Personalized and heterogeneous FL
FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedLMT, a federated learning framework where all clients train the same pre-factorized low-rank model (hybrid architecture) to handle system heterogeneity without heterogeneous aggregation, with a convergence proof linking low-rank training to full-model stationarity, plus pFedLMT for personalization.
Problem
Resource-constrained clients in federated learning cannot train the full model, and aggregating heterogeneous sub-models degrades global accuracy; most existing sub-model methods lack convergence analysis.
Approach
Assign every client the same pre-factorized low-rank model (hybrid architecture with spectral init and Frobenius/L2/Kronecker regularization) so no heterogeneous aggregation is needed, and prove that stationarity of the low-rank model implies stationarity of the recovered full model; extend to pFedLMT by keeping custom factorized layers local.
Key results
- On CIFAR10 under β4-β3-β2 settings, FedLMT reaches 91.03% accuracy vs 91.91% for FedAvg while cutting communication from 223.5 GB to 28.62 GB and FLOPs from 11.18e12 to 2.80e12.
- On WikiText2, FedLMT achieves perplexity 2.93 with 2.65 GB communication, versus 3.52 for FedAvg at 10.36 GB.
- On TINY, FedLMT attains 48.53% accuracy versus 42.71% for FedAvg under the same heterogeneous setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a homogeneous low-rank model with FedLMT instead of FedAvg with full resource availability | Federated ResNet-18 · CIFAR10, β4 - β3 - β2 | top-1 test accuracy: 91.91 -> 91.03 | Hurt |
| Train a homogeneous low-rank model with FedLMT instead of FedAvg with full resource availability | Federated ResNet-18 · CIFAR100, β4 - β3 - β2 | top-1 test accuracy: 72.20 -> 71.08 | Hurt |
| Train a homogeneous low-rank model with FedLMT instead of FedRolex | Federated ResNet-18 · SVHN, β4 - β3 - β2 | top-1 test accuracy: 94.62 -> 95.35 | Helped |
| Train a homogeneous low-rank model with FedLMT instead of DepthFL | Federated ResNet-18 · Tiny-ImageNet, β4 - β3 - β2 | top-1 test accuracy: 44.84 -> 48.53 | Helped |
| Train a homogeneous low-rank model with FedLMT instead of HeteroFL | Federated Transformer · WikiText2 | perplexity: 3.06 -> 2.93 | Helped |
| Add L2 decay (ablation) | Low-rank ResNet-18 · CIFAR10 | top-1 test accuracy: 91.27 -> 91.76 | Helped |
| Add Frobenius decay (ablation) | Low-rank ResNet-18 · CIFAR10 | top-1 test accuracy: 91.27 -> 92.67 | Helped |
| Add Kronecker decay (ablation) | Low-rank ResNet-18 · CIFAR10 | top-1 test accuracy: 91.27 -> 91.99 | Helped |
| Add L2 decay and Kronecker decay (ablation) | Low-rank ResNet-18 · CIFAR10 | top-1 test accuracy: 91.27 -> 92.60 | Helped |
| Add L2 decay and Frobenius decay (ablation) | Low-rank ResNet-18 · CIFAR10 | top-1 test accuracy: 91.27 -> 92.91 | Helped |
| Add Kronecker decay and Frobenius decay (ablation) | Low-rank ResNet-18 · CIFAR10 | top-1 test accuracy: 91.27 -> 92.70 | Helped |
| Increase the number of undecomposed layers in the hybrid model (ablation) | Low-rank ResNet-18 · CIFAR10 and CIFAR100 | test accuracy | Helped |
Surprising findings
- Training only the smallest homogeneous low-rank model (β2) can outperform heterogeneous-model baselines trained under mixed capacities, suggesting heterogeneous sub-model aggregation harms performance.
Limitations
- Physical implications of the low-rank connection between low-rank and full model remain unclear.
- Theorem 4.5 does not reflect how smallest singular value and low-rank ratio affect convergence rate.
- Assumption 4.4 (smallest singular value positive) is needed; if violated the learning rate must be zero.
Experimental setup
Datasets: SVHN CIFAR10 CIFAR100 Tiny-ImageNet WikiText2
Benchmarks: SVHN CIFAR10 CIFAR100 Tiny-ImageNet WikiText2
Metrics: top-1 test accuracy perplexity communication cost (GB) FLOPs
Baselines: FedAvg FedDropout HeteroFL FedRolex DepthFL ProgFed FedHM FLANC FedRep
Builds on: FedAvg FedHM Pufferfish HeteroFL FedRep
Closest papers by meaning
- Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model (AAAI 2025 · similarity 0.92)
- FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity (ICLR 2024 · similarity 0.91)
- Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.91)
- A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings (NeurIPS 2024 · similarity 0.91)
- Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneity (AAAI 2024 · similarity 0.90)
- One Arrow, Two Hawks: Sharpness-aware Minimization for Federated Learning via Global Model Trajectory (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.