AI Paper Atlas

Research map / Federated learning and privacy / Personalized and heterogeneous FL

FedLoVA: Value-Only Aggregation for Federated LoRA Fine-Tuning of Large Language Models

Ensieh Khazaei, Baturalp Buyukates, Dimitrios Hatzinakos

NeurIPS 2026 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated prompt and LoRA tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FedLoVA, a federated LoRA fine-tuning method that aggregates only the value-projection LoRA components, motivated by analysis of the distinct roles of query and value projections.

Problem

LoRA is applied to both query and value projections in transformers in federated learning without considering their distinct roles, which affects federated LoRA fine-tuning of LLMs.

Approach

Analyzes the distinct roles of LoRA on query versus value projections and proposes value-only aggregation in federated LoRA, so that only value-projection adapters are aggregated across clients.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
share only value-projection LoRA instead of both Q and VFedLoRA · GLUE avg (RoBERTa-large)accuracy: 87.69 -> 88.13Improved
share V rather than Q with server (ablation)FedLoVA vs FedLoQA · GLUE avgaccuracy: 86.89 -> 88.13Improved
share only V (halving communication)FedLoRA · communication cost to target accuracytotal comm.: 215.3 M -> 68.3 M (MNLI-m)Improved
apply LoRA to V instead of Q (ablation)LoRA · SST2/MNLI-mm convergence (RoBERTa-large)rounds to target: 3.3x / 2.4x fewer roundsImproved
FedLoVA vs FedLoRA on multilingual sentimentFedLoRA · Multilingual Sentimentaccuracy: 68.90 -> 70.18Improved
FedLoVA vs FedLoRA on GSM8KFedLoRA · GSM8K (Llama3-8B)accuracy: 43.59 -> 45.79Improved
share Q only instead of V (ablation)FedLoVA · GSM8K (Llama3-8B)accuracy: 45.79 -> 38.74Hurt
add FedLoVA over FFA-LoRA on RTEFFA-LoRA · RTEaccuracy: 80.99 -> 84.13Improved
add FedLoVA over FedSA-LoRA on QQPFedSA-LoRA · QQPaccuracy: 82.82 -> 86.23Improved
apply LoRA to Q+K+V instead of Q+V (ablation)LoRA · MNLI-mmaccuracy: 87.80 -> 88.03No clear effect
adapt MLP instead of Q/V (ablation)LoRA · SST2accuracy: 95.49 -> 95.70Mixed
add FedLoVA vs FedLoRA with 50 clientsFedLoRA baselines · MNLI-m, 50 clientsaccuracy: 88.60 -> 89.18Improved

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Experimental setup

Builds on: LoRA

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy