Research map / Federated learning and privacy / Personalized and heterogeneous FL
FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FLoRA, a federated LoRA fine-tuning method that aggregates client LoRA adapters by stacking them, giving noise-free aggregation and supporting heterogeneous ranks across clients.
Problem
Federated fine-tuning of LLMs with LoRA: naive independent averaging of LoRA A and B matrices (FedIT) introduces aggregation noise, and FedIT cannot aggregate clients with heterogeneous LoRA ranks.
Approach
Stack the clients' A_k matrices vertically and B_k matrices horizontally so that the product of the stacked matrices exactly equals the sum of client updates sum_k p_k B_k A_k, removing cross-client intermediate noise and permitting different ranks per client.
Key results
- In the homogeneous setting on TinyLlama with Dolly, FLoRA reaches 30.80 MMLU vs. 16.35 for FedIT.
- In the heterogeneous setting on Llama with Alpaca, zero-padding drops MMLU to 7.97 while FLoRA reaches 29.54.
- On Llama, FLoRA with heterogeneous ranks achieves MT-bench 4.14 on Wizard vs. 3.51 for zero-padding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace FedIT's independent LoRA averaging with stacking | Federated LoRA fine-tuning · Homogeneous LoRA; MMLU and MT-bench across TinyLlama and Llama | accuracy / score: 16.35 -> 30.80; 30.02 -> 31.92; 42.51 -> 43.87; 2.92 -> 3.13; 2.55 -> 2.77; 29.67 -> 30.99; 29.41 -> 29.85; 33.43 -> 34. | Helped |
| Use stacking rather than zero-padding for heterogeneous LoRA ranks | Federated LoRA fine-tuning · Heterogeneous LoRA; MMLU and MT-bench across TinyLlama and Llama | accuracy / score: 15.76 -> 18.45; 29.56 -> 29.69; 40.79 -> 41.48; 1.56 -> 3.14; 1.29 -> 2.71; 26.46 -> 28.50; 7.97 -> 29.54; 26.98 -> 27.9 | Helped |
| Vary the scaling factor among 0.01, 0.05, 0.1, and 0.2 (ablation) | FLoRA · Dolly and Wizard; TinyLlama and Llama | accuracy | Mixed |
| Use scaling factor 0.01 instead of 0.1 or 0.2 (ablation) | FLoRA with TinyLlama · Dolly | accuracy | Helped |
| Use scaling factor 0.01 instead of 0.2 (ablation) | FLoRA with TinyLlama · Wizard | accuracy | Hurt |
| Use larger scaling factors (ablation) | FLoRA with Llama · Dolly and Alpaca | accuracy | Helped |
| Vary client LoRA rank (ablation) | FLoRA · MT-bench across models and datasets | score | Mixed |
| Apply AdaLoRA rank adaptation (ablation) | FLoRA with TinyLlama · MT-bench | score: 3.13 -> 3.14 | Helped |
| Apply AdaLoRA rank adaptation (ablation) | FLoRA with Llama · MT-bench | score: 4.21 -> 4.10 | Hurt |
| Apply AdaLoRA rank adaptation (ablation) | FLoRA with Llama2 · MT-bench | score: 4.17 -> 4.25 | Helped |
Surprising findings
- On TinyLlama with Alpaca and Wizard, FLoRA in the federated setting exceeds centralized LoRA fine-tuning, which the authors attribute to reduced overfitting.
Limitations
- Stacked global LoRA modules increase server-to-client communication relative to FedIT.
- Only Llama-family models (TinyLlama, Llama, Llama2) were evaluated.
- Experiments use few communication rounds (1 to 3) due to compute constraints, and MMLU was evaluated on a 1444-question subset.
Experimental setup
Datasets: Databricks-dolly-15k Alpaca WizardLM training data ShareGPT
Benchmarks: MMLU MT-bench
Metrics: accuracy MT-bench score (GPT-4 judged)
Baselines: FedIT Zero-padding Centralized LoRA Standalone
Builds on: LoRA FedAvg FedIT AdaLoRA
Closest papers by meaning
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.92)
- Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes (ICML 2024 · similarity 0.91)
- Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources (NeurIPS 2024 · similarity 0.91)
- Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.90)
- Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.