Research map / Federated learning and privacy / Personalized and heterogeneous FL
Selective Aggregation for Low-Rank Adaptation in Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that in federated LoRA the A matrices learn general knowledge while B matrices capture client-specific knowledge, and proposes FedSA-LoRA, which shares only A and keeps B local, extended to rsLoRA and VeRA variants.
Problem
Aggregating LoRA A and B matrices separately in federated learning introduces aggregation errors, while freezing A (FFA-LoRA) impairs learning and enforces a uniform model update across non-IID clients.
Approach
Asymmetry analysis shows A matrices learn shared general knowledge and B matrices learn client-specific knowledge; so only A is aggregated on the server while B stays local, enabling personalized updates while keeping both matrices trainable.
Key results
- FedSA-LoRA reaches 90.43 average accuracy on GLUE with RoBERTa-large vs 89.33 for LoRA and 89.39 for FFA-LoRA.
- FedSA-LoRA needs 91 communication rounds on RTE vs 167 for LoRA and 229 for FFA-LoRA, with half the per-round communicated parameters of LoRA.
- On GSM8K with LLaMA3-8B, FedSA-LoRA scores 46.63 vs 46.23 for LoRA and 46.32 for FFA-LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedSA-LoRA instead of FedDPA-LoRA | LoRA with federated RoBERTa-large · MNLI-m | accuracy: 88.99 -> 90.18 | Helped |
| Use FedSA-LoRA instead of FedDPA-LoRA | LoRA with federated RoBERTa-large · QNLI | accuracy: 90.74 -> 92.13 | Helped |
| Use FedSA-rsLoRA instead of FedDPA-rsLoRA | rsLoRA with federated RoBERTa-large · RTE | accuracy: 86.26 -> 88.00 | Helped |
| Use FedSA-VeRA instead of FedDPA-VeRA | VeRA with federated RoBERTa-large · RTE | accuracy: 86.12 -> 87.83 | Helped |
| Keep B local and aggregate A rather than using FFA-LoRA | LoRA with federated LLaMA3-8B · GSM8K | accuracy: 46.32 -> 46.63 | Helped |
| Keep B local and aggregate A rather than using FFA-LoRA | LoRA with federated LLaMA3-8B · CodeSearchNet | performance score: 58.57 -> 59.66 | Helped |
| Use FedSA-LoRA rather than LoRA | LoRA with federated clients · QNLI, severe non-IID (Split-3) | accuracy: 90.60 -> 92.00 | Helped |
| Use FedSA-LoRA rather than LoRA | LoRA with federated clients · MNLI-m, severe non-IID (Split-3) | accuracy: 88.80 -> 90.20 | Helped |
| Increase LoRA rank from r=2 to r=16 (ablation) | FedSA-LoRA · QNLI | accuracy: 92.69 -> 92.03 | Hurt |
| Increase LoRA rank from r=2 to r=16 (ablation) | FedSA-LoRA · SST2 | accuracy: 95.67 -> 95.78 | Mixed |
| Increase LoRA rank from r=2 to r=16 (ablation) | FedSA-LoRA · MNLI-m | accuracy: 89.14 -> 89.59 | Helped |
Surprising findings
- Fixing A (FFA-LoRA) does not outperform standard LoRA on most GLUE tasks, contrary to the expectation that it is a strong communication-efficient baseline.
Limitations
- Experiments mainly use a small number of clients (three) on GLUE with RoBERTa-large; generation experiments are limited to IID splits for GSM8K.
- Convergence analysis relies on strong assumptions (L-smoothness, bounded gradients and eigenvalue conditions).
- VeRA-based variants required a different optimizer (AdamW) after SGD performed poorly, complicating comparison.
Experimental setup
Datasets: GLUE (MNLI, SST2, QNLI, QQP, RTE) GSM8K CodeSearchNet
Benchmarks: GLUE GSM8K CodeSearchNet
Metrics: accuracy communication rounds to target trainable parameters per-round communicated parameters
Baselines: LoRA FFA-LoRA FedDPA-LoRA rsLoRA FFA-rsLoRA FedDPA-rsLoRA VeRA FFA-VeRA FedDPA-VeRA
Builds on: LoRA rsLoRA VeRA FFA-LoRA FedAvg
Closest papers by meaning
- Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA (NeurIPS 2025 · similarity 0.91)
- FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA (NeurIPS 2025 · similarity 0.90)
- Improving LoRA in Privacy-preserving Federated Learning (ICLR 2024 · similarity 0.88)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.88)
- Closed-Form Merging of Parameter-Efficient Modules for Federated Continual Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.