Research map / Federated learning and privacy / Differential privacy
FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedSVD, which periodically re-initializes the LoRA A matrix with orthonormal right singular vectors from an SVD of the aggregated BA product, so that only B is trained under DP-SGD in federated learning.
Problem
LoRA combined with DP-SGD in federated learning suffers noise amplification through the BA product, and freezing A (as in FFA-LoRA) limits expressiveness and slows convergence.
Approach
After each aggregation, the server refactorizes B·A via SVD and sets A to the top-r orthonormal right singular vectors and B to U·Σ, so A adapts to aggregated updates without being noised, and orthonormal A bounds gradient norms of B under clipping.
Key results
- Under DP with epsilon=6, FedSVD averages 76.79% across six GLUE tasks versus 68.02% for FFA-LoRA (+8.77 pp).
- Under DP with epsilon=3, FedSVD averages 75.63%, a +9.63 pp gain over FFA-LoRA (66.00%).
- Without privacy, FedSVD averages 86.18% versus 84.57% for FFA-LoRA (+1.29 pp); on HellaSwag it reaches 51.10% versus 49.76% for FFA-LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed random A with periodic SVD-based adaptive A | LoRA federated fine-tuning · GLUE average, non-private | accuracy: +1.29 percentage points | Helped |
| replace fixed random A with periodic SVD-based adaptive A | LoRA federated fine-tuning · GLUE average, DP ε=6 | accuracy: +8.77 pp | Helped |
| replace fixed random A with periodic SVD-based adaptive A | LoRA federated fine-tuning · GLUE average, DP ε=3 | accuracy: +9.63 pp | Helped |
| replace FFA-LoRA with FedSVD | LoRA federated fine-tuning · HellaSwag, DP ε=6 | accuracy: +1.34 pp | Helped |
| add orthonormal rows to periodically reinitialized A (ablation) | FedSVD · MNLI matched and mismatched, DP ε=6 | accuracy: +0.92/+1.17 pp | Helped |
| add orthonormal rows to periodically reinitialized A (ablation) | FedSVD · SNLI | accuracy: +11.68 pp | Helped |
| use fixed random orthonormal A instead of unstructured fixed A (ablation) | FFA-LoRA · MNLI matched and mismatched, DP ε=6 | accuracy | No clear effect |
| initialize fixed A and B using PiSSA (ablation) | FFA-LoRA · MNLI matched and mismatched, DP ε=6 | accuracy | Helped |
| replace full SVD with low-rank SVD approximation (ablation) | FedSVD · SNLI, MNLI matched, SST-2, QQP and QNLI, DP ε=6 | accuracy: 9.12 ± 0.08; 0.15 ± 0.04; 0.89 ± 0.07 | Mixed |
| reduce SVD reinitialization frequency to every 2, 5 or 10 rounds (ablation) | FedSVD · MNLI, DP ε=6 | accuracy | No clear effect |
| integrate DoRA instead of LoRA (ablation) | FedSVD · MNLI matched and mismatched | accuracy | No clear effect |
| increase total clients while keeping 3 participating per round (ablation) | FedSVD · MNLI, DP ε=6 | accuracy | Mixed |
Surprising findings
- FLoRA and FedEX-LoRA perform worse than FedAvg and FFA-LoRA in both private and non-private settings, attributed to frequent random re-initialization.
- Removing the orthonormal constraint on A reduces SNLI accuracy from 81.16% to 69.48%, a gap of nearly 12 pp, though the MNLI gap is small (about 1 pp).
Limitations
- Server-side SVD adds computational overhead, mitigated by randomized low-rank SVD.
- Broadcasting the new A adds communication cost unless SVD is decentralized to clients.
- Extreme heterogeneity (Dirichlet alpha=0.1) causes all methods to fail; the theory covers only a two-layer ReLU MLP.
Experimental setup
Datasets: SNLI MNLI SST-2 QQP QNLI HellaSwag
Benchmarks: SNLI MNLI SST-2 QQP QNLI HellaSwag
Metrics: accuracy 95% confidence interval
Baselines: FedAvg FFA-LoRA FLoRA FedEX-LoRA PiSSA
Builds on: LoRA DP-SGD FFA-LoRA FedAvg PiSSA
Closest papers by meaning
- Improving LoRA in Privacy-preserving Federated Learning (ICLR 2024 · similarity 0.92)
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix (NeurIPS 2025 · similarity 0.92)
- Selective Aggregation for Low-Rank Adaptation in Federated Learning (ICLR 2025 · similarity 0.90)
- Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning (NeurIPS 2026 · similarity 0.89)
- Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.