Research map / Federated learning and privacy / Communication-efficient FL
Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoLoRA, a federated LoRA fine-tuning method that alternates between freezing and aggregating the down-projection (A) and up-projection (B) matrices, with linear-model theory showing it beats FFA-LoRA and experiments on RoBERTa-Large and Llama-2-7B.
Problem
Naive FedAvg on LoRA's A and B matrices yields inexact aggregated updates (average of A_i B_i differs from product of averages), while freezing one of the matrices (FFA-LoRA) limits expressiveness and robustness, especially with fewer fine-tuning parameters or more clients.
Approach
Alternate between odd and even communication rounds: in one round all clients freeze A and train/aggregate B, in the next they freeze B and train/aggregate A, so both down- and up-projections are learned while keeping aggregation exact and halving per-round communication. Theory on a rank-1 linear model shows exponential convergence versus FFA-LoRA's error floor set by initialization.
Key results
- On RoBERTa-Large with 50 clients (rank 4), RoLoRA averages 85.81 across GLUE tasks versus 70.72 for LoRA, 76.48 for FFA-LoRA and 54.67 for FlexLoRA.
- With 3 clients (rank 4), RoLoRA averages 88.28 versus 88.20 for LoRA and 87.48 for FFA-LoRA.
- RoLoRA reduces per-round communication by half relative to LoRA and FlexLoRA while matching or exceeding LoRA accuracy in high-client settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use alternating updates of both LoRA projection matrices rather than directly averaging LoRA matrices | RoBERTa-Large federated finetuning · GLUE average, 50 clients, rank 4 | accuracy: 76.48 -> 85.81 | Helped |
| Use alternating updates of both LoRA projection matrices rather than directly averaging LoRA matrices | RoBERTa-Large federated finetuning · MNLI, Dir(1.0), 15 clients | accuracy: 74.54 -> 81.55 | Helped |
| Increase clients from 3 to 50 while maintaining the sample count | LoRA · GLUE average, rank 4 | accuracy: 88.20 -> 70.72 | Hurt |
| Learn projection A instead of keeping it frozen (ablation) | FFA-LoRA / RoLoRA · MNLI, 50 clients | accuracy | Helped |
| Prioritize multiple B updates over balanced A/B alternation (ablation) | RoLoRA · Accuracy over communication rounds | accuracy | Hurt |
| Prioritize multiple A updates over balanced A/B alternation (ablation) | RoLoRA · Accuracy over communication rounds | accuracy | Hurt |
| Use unequal learning rates for A and B instead of balanced alternation (ablation) | RoLoRA · Accuracy over communication rounds | accuracy | Hurt |
| Increase FFA-LoRA local steps per round while keeping total computation fixed (ablation) | FFA-LoRA · MNLI, 50 clients | accuracy: 72.52 -> 71.73 -> 69.64 -> 69.97 | Hurt |
| Increase FFA-LoRA local steps per round while keeping total computation fixed (ablation) | FFA-LoRA · QQP, 50 clients | accuracy: 80.51 -> 80.2 -> 79.07 -> 78.44 | Hurt |
| Vary RoLoRA local steps per round while keeping total computation fixed (ablation) | RoLoRA · MNLI, 50 clients | accuracy: 84.39 -> 84.96 -> 84.79 -> 82.98 | No clear effect |
| Vary RoLoRA local steps per round while keeping total computation fixed (ablation) | RoLoRA · QQP, 50 clients | accuracy: 85.24 -> 85.44 -> 84.77 -> 85.71 | No clear effect |
Surprising findings
- FlexLoRA (matrix multiplication plus truncated SVD aggregation) collapses to about 54.67 average accuracy at 50 clients, below FFA-LoRA.
- FFA-LoRA shows very large variance across seeds on commonsense tasks (std 9.55 on PIQA), attributed to initialization of the frozen A.
Limitations
- Theory covers linear (rank-1) models with full client participation; non-linear and partial participation are not proven.
- Convergence analysis for non-convex losses follows standard FL rates and does not cover LoRA specifically.
Experimental setup
Datasets: SST-2 QNLI MNLI QQP RTE MNLI (MNIST-free text partitions) HumanEval MMLU commonsense reasoning tasks (8 tasks incl. PIQA, SIQA)
Benchmarks: GLUE (SST-2, QNLI, MNLI, QQP, RTE) commonsense reasoning (8 tasks) HumanEval MMLU MNIST (toy non-linear model)
Metrics: accuracy standard deviation over seeds
Baselines: LoRA (FedAvg of LoRA) FFA-LoRA FlexLoRA FLoRA
Builds on: LoRA FedAvg FFA-LoRA FedRep FederatedScope-LLM
Closest papers by meaning
- Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.93)
- Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning (NeurIPS 2026 · similarity 0.93)
- Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.92)
- Selective Aggregation for Low-Rank Adaptation in Federated Learning (ICLR 2025 · similarity 0.91)
- FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.90)
- Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.