Research map / Federated learning and privacy / Personalized and heterogeneous FL
Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlexLoRA, a federated LoRA aggregation scheme that averages full-size client LoRA products, re-decomposes them with SVD, and redistributes rank-truncated factors so clients can use heterogeneous ranks matched to their resources.
Problem
Federated fine-tuning of LLMs with LoRA suffers when clients have heterogeneous data and compute resources; standard aggregation restricts all clients to the lowest rank (the "bucket effect"), hurting generalization to unseen clients and tasks.
Approach
Clients train LoRA at their own rank; the server composes each client's full weight product sBA, takes a weighted average, and re-decomposes the global update with SVD, sending each client its top-r^i singular components, so no hyperparameters are needed for aggregation.
Key results
- FlexLoRA improves weighted-average zero-shot Rouge-L over homogeneous-rank baselines, with average gains of 2.14% for FedAvg, 0.86% for FedIT and 1.94% for SLoRA.
- FlexLoRA consistently outperforms HETLORA across all four resource distributions in the unseen-client Rouge-L evaluation (e.g., FedAvg Uniform 58.07 vs 56.85).
- On Dolly-15K with DataJuicer-1.3B, the Uniform resource distribution reduces total cost to about 33.1% of homogeneous rank (66.9% reduction) while converging in fewer rounds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate FlexLoRA with homogeneous-rank FedAvg, FedIT, or SLoRA | FedAvg, FedIT, and SLoRA · Unseen clients across resource distributions | Rouge-L: 2.14\% for FedAvg, 0.86\% for FedIT, and 1.94\% for SLoRA | Helped |
| Use FlexLoRA instead of HETLORA for heterogeneous-rank aggregation | Heterogeneous-rank FL methods · Unseen clients across all resource distributions | Rouge-L | Helped |
| Integrate FlexLoRA with FedIT at homogeneous rank 8 (ablation) | FedIT · Test loss convergence, homogeneous rank 8 | test loss | No clear effect |
| Increase homogeneous LoRA rank from 8 to 200 (ablation) | FedIT · Test loss convergence, homogeneous rank | test loss | Helped |
| Apply LoRA to all linear layers instead of attention layers only (ablation) | FedIT and FedIT + FlexLoRA · Unseen-client zero-shot evaluation, Uniform resources | Rouge-L: 0.5701 -> 0.6195; 0.5751 -> 0.6211 | Helped |
| Increase homogeneous LoRA rank from 8 to 200 (ablation) | FedIT · Single-client task examples | performance: 0.8977 -> 0.9157; 0.9084 -> 0.9272; 0.549 -> 0.5749; 0.5031 -> 0.5301; 0.4414 -> 0.4507; 0.5173 -> 0.5328; 0.6995 -> 0.7 | Helped |
| Increase rank using heterogeneous resources rather than homogeneous rank (ablation) | FedAvg · Dolly-15K, mixed tasks, DataJuicer (1.3B) | Rouge-L: $55.34 \pm 0.59$ -> $58.56 \pm 1.35$ | Helped |
| Increase rank using heterogeneous resources rather than homogeneous rank (ablation) | FedAvg · Dolly-15K, mixed tasks, LLaMA-3 (8B) | Rouge-L: $64.26 \pm 0.80$ -> $64.93 \pm 0.33$ | Helped |
| Use heterogeneous ranks with SLoRA on Heavy-Tail-Light resources | SLoRA + FlexLoRA · Cross-task natural language understanding | task performance | Hurt |
| Increase the participating client pool (ablation) | FedAvg + FlexLoRA · Dolly-15K, varied client counts | generalization | Helped |
| Use heterogeneous ranks rather than homogeneous rank (ablation) | FedAvg · Dolly-15K, convergence to loss 2 | total FL cost: Heavy-Tail-Light: total reduction of 49.4\%; Uniform: 66.9\% | Helped |
| Integrate FlexLoRA with FedAvg on Heavy-Tail-Light resources (ablation) | FedAvg · Dolly-15K, mixed tasks, DataJuicer (1.3B) | Rouge-L: $55.34 \pm 0.59$ -> $58.39 \pm 1.26$ | Helped |
Surprising findings
- Under FedIT, FlexLoRA underperforms on word-level analysis tasks such as word analogy and keyword tagging, which the authors attribute to Adam momentum overfitting in the top decoding block.
- SLoRA with the heavy-tail-light distribution is an exception where FlexLoRA does not improve, possibly due to the dominance of rank-8 clients.
Limitations
- LLaMA-3 8B was not tested in the thousands-of-clients setting due to resource limits.
- FlexLoRA gains are modest (around 1-2% average) and not significant in all settings per the reported p-values.
- Theoretical bound relies on Lipschitz and SVD-error assumptions that may not hold exactly in practice.
Experimental setup
Datasets: Natural Instructions Dolly-15K
Benchmarks: Natural Instructions (unseen clients, Rouge-L) Natural Instructions English evaluation track (12 categories, 119 tasks) Dolly-15K
Metrics: Rouge-L test loss percentage improvement p-value
Baselines: FedAvg FedIT SLoRA HETLORA
Builds on: LoRA FedAvg FedIT HETLORA SLoRA
Closest papers by meaning
- FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.91)
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.91)
- Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.90)
- Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA (NeurIPS 2025 · similarity 0.90)
- Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.88)
- Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.