Research map / Federated learning and privacy / Personalized and heterogeneous FL
Improving LoRA in Privacy-preserving Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FFA-LoRA, which freezes the randomly initialized A matrix and trains only the zero-initialized B matrix, to make LoRA more stable and private/communication-efficient in federated learning with differential privacy.
Problem
LoRA becomes unstable in privacy-preserving federated learning due to data heterogeneity with multi-step local updates, noise amplification from DP-SGD on the product of two low-rank matrices, and sensitivity to the scaling factor alpha.
Approach
Freeze the randomly initialized low-rank matrix A and train only the zero-initialized matrix B, so the federated aggregate equals the ideal averaged update, DP noise is not cascaded through a product of two noisy matrices, and alpha becomes unnecessary (equivalent to LoRA with alpha tending to infinity).
Key results
- Under DP with epsilon=1 on MNLI (severe heterogeneity, 3 clients), FFA-LoRA reaches 75.05% vs 33.80% for LoRA with RoBERTa-Large.
- On GSM8K with LLaMA-7B, FFA-LoRA achieves 17.12% accuracy versus 15.68% for LoRA.
- Without privacy, FFA-LoRA at rank 16 has 1.57M trainable parameters (0.44%), half of LoRA rank 16, while matching or beating LoRA on most tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LoRA with FFA-LoRA by freezing A and training B | RoBERTa-Large · MNLI matched, ε = 6 | accuracy (%): 39.46_{\pm 14.3 } -> 78.81_{\pm 0.8 } | Helped |
| Replace LoRA with FFA-LoRA by freezing A and training B | LLaMA-7B · GSM8K | accuracy (%): 15.68\% -> 17.12\% | Helped |
| Replace LoRA with FFA-LoRA by freezing A and training B | RoBERTa-Large · SST-2, ε = 6 | accuracy (%): 93.70_{\pm 0.5} -> 93.73_{\pm 0.3} | No clear effect |
| Use LR scaling with α = 2 instead of the α = 8 baseline (ablation) | LoRA · QNLI | accuracy (%): 91.36\% -> 71.88\% | Hurt |
| Use LR scaling with α = 64 instead of the α = 8 baseline (ablation) | LoRA · QNLI | accuracy (%): 91.36\% -> 50.96\% | Hurt |
| Vary α with learning-rate scaling (ablation) | FFA-LoRA · QNLI | accuracy (%): 91.31\% (α = 2) to 92.46\% (α = 256) | No clear effect |
| Increase rank from 8 to 16 (ablation) | FFA-LoRA · QNLI, non-private | accuracy (%): 91.62 -> 91.65 | No clear effect |
| Reduce rank from 8 to 2 (ablation) | LoRA · QNLI, ε = 1 | accuracy (%): 81.45 -> 58.15 | Hurt |
| Reduce rank from 8 to 2 (ablation) | FFA-LoRA · QNLI, ε = 1 | accuracy (%): 83.01 -> 82.64 | No clear effect |
| Replace Kaiming A initialization with orthogonal initialization (ablation) | FFA-LoRA · QNLI | accuracy (%): 91.84\% -> 92.16\% | No clear effect |
| Replace LoRA with QVP adapters (ablation) | RoBERTa-Large · QNLI, ε = 6, matched parameter budget | accuracy (%): 87.33 \% -> 86.41\% | Hurt |
| Replace LoRA with FFA-LoRA | Vision Transformer · Food-101, centralized | accuracy: 86.18\% -> 85.83\% | No clear effect |
Surprising findings
- Under strong DP (epsilon=1) and severe heterogeneity, vanilla LoRA collapses to near-chance accuracy on MNLI (about 34%) while the frozen-A variant retains about 75%.
- Freezing A and training only B, a method with half the trainable parameters, matches or outperforms full LoRA even without privacy.
Limitations
- Theoretical smoothness and equivalence results hold under simplifying assumptions (locally linear model, small learning rate).
- Results on GSM8K and vision are limited to single runs or small set of experiments; the paper notes the advantage is not significant in some settings (e.g., centralized vision).
Experimental setup
Datasets: MNLI SST-2 QQP QNLI GSM8K Food-101
Benchmarks: MNLI SST-2 QQP QNLI GSM8K Food-101
Metrics: accuracy
Baselines: LoRA full fine-tuning QVP adapters
Builds on: LoRA FedAvg DP-SGD
Closest papers by meaning
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix (NeurIPS 2025 · similarity 0.92)
- FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA (NeurIPS 2025 · similarity 0.92)
- Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning (NeurIPS 2026 · similarity 0.90)
- Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA (NeurIPS 2025 · similarity 0.89)
- Privacy-Preserving Low-Rank Adaptation Against Membership Inference Attacks for Latent Diffusion Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.