Research map / Federated learning and privacy / Personalized and heterogeneous FL
Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ravan, a federated LoRA method that sums multiple frozen-basis LoRA heads with trainable scaling factors and core matrices, enabling higher effective rank and exact aggregation under data and computational heterogeneity.
Problem
Federated fine-tuning of LLMs with LoRA suffers accuracy degradation under data heterogeneity and computational heterogeneity, and existing fixes either lose exact aggregation, add communication cost, or cannot raise effective rank within a parameter budget.
Approach
Reparameterize weight updates as a sum of h heads s_i B_i H_i A_i where B_i and A_i are frozen at initialization with orthogonal column/row spaces (Gram-Schmidt or random normal), only H_i and scalar s_i are trained and communicated as s_i H_i, so effective rank grows as sqrt(Nh) with exact aggregation and no extra communication; clients can freeze heads to match their compute budget.
Key results
- On CIFAR-100 with 50 non-I.I.D. clients at the lower budget, Ravan reaches 73.80% vs 66.58% for FedEx-LoRA and 68.15% for FedIT.
- Ravan achieves the best result in all vision configurations and 11 of 16 language configurations among PEFT methods.
- On GLUE with LLaMA3.2-1B (20 clients, lower budget), Ravan averages 83.90 vs 83.21 for Fed-SB, a 0.7-point gain over the next best baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Ravan's multi-head update instead of Fed-SB | ViT-B/16 · CIFAR-100, 20 clients, non-I.I.D., lower budget | accuracy (%): 71.48 -> 76.22 | Helped |
| use Ravan's multi-head update instead of Fed-SB | T5-Base · 20 Newsgroups, 20 clients, non-I.I.D., lower budget | accuracy (%): 63.10 -> 65.73 | Helped |
| use Ravan's multi-head update instead of strongest baseline FedEx-LoRA | T5-Base · MRQA, 20 clients, non-I.I.D., lower budget | F1 (%): 60.68 -> 60.45 | Hurt |
| use Gram-Schmidt rather than random normal initialization (ablation) | Ravan · CIFAR-100 and SVHN, 20 non-I.I.D. clients, lower budget | accuracy (%): 76.22 -> 78.75; 90.02 -> 91.25 | Mixed |
| use random normal rather than Gram-Schmidt initialization (ablation) | Ravan · 20 Newsgroups and MRQA, 20 non-I.I.D. clients, lower budget | accuracy (%) / F1 (%): 64.83 -> 65.73; 59.71 -> 60.45 | Mixed |
| use constant shared bases rather than random normal initialization (ablation) | Ravan · CIFAR-100, SVHN, 20 Newsgroups, and MRQA, 20 non-I.I.D. clients, lower budget | accuracy (%) / F1 (%): 76.22 -> 58.12; 90.02 -> 88.01; 65.73 -> 56.74; 60.45 -> 60.43 | Hurt |
| initialize heads in a shared subspace instead of with random normal bases (ablation) | Ravan · CIFAR-100, SVHN, 20 Newsgroups, and MRQA, 20 non-I.I.D. clients, lower budget | accuracy (%) / F1 (%): 76.22 -> 57.39; 90.02 -> 84.54; 65.73 -> 55.64; 60.45 -> 60.19 | Hurt |
| train per-head scaling factors instead of fixing them to 1 (ablation) | Ravan · CIFAR-100, 20 clients, non-I.I.D., lower budget | accuracy (%): 74.93 -> 76.22 | Helped |
| train per-head scaling factors instead of fixing them to 1 (ablation) | Ravan · SVHN, 20 clients, non-I.I.D., lower budget | accuracy (%): 90.35 -> 90.02 | Hurt |
| train per-head scaling factors instead of fixing them to 1 (ablation) | Ravan · 20 Newsgroups and MRQA, 20 clients, non-I.I.D., lower budget | accuracy (%) / F1 (%): 65.24 -> 65.73; 60.45 -> 60.60 | Helped |
| increase the number of heads beyond the point where effective rank saturates (ablation) | Ravan · Vision and language benchmarks | accuracy | Hurt |
| use weight-based rather than random or gradient-based head scoring (ablation) | Ravan · Computational heterogeneity benchmarks | accuracy | Hurt |
Surprising findings
- Weight-based head scoring consistently lags random and gradient-based scoring because it always selects the same high-magnitude heads across clients.
- Constant (non-trainable) scaling factors slightly outperform trainable ones on SVHN and MRQA-like settings, though trainable scaling wins on three of four datasets.
Experimental setup
Datasets: CIFAR-100 SVHN 20 Newsgroups MRQA GLUE
Benchmarks: CIFAR-100 SVHN 20 Newsgroups MRQA GLUE
Metrics: accuracy F1
Baselines: Full-FT FedIT FedEx-LoRA FFA-LoRA Fed-SB HetLoRA FlexLoRA
Builds on: LoRA LoRA-XS HydraLoRA Fed-SB FedIT
Closest papers by meaning
- Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA (NeurIPS 2025 · similarity 0.92)
- Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.90)
- Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning (NeurIPS 2026 · similarity 0.90)
- Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes (ICML 2024 · similarity 0.90)
- Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.