AI Paper Atlas

Research map / Federated learning and privacy / Personalized and heterogeneous FL

Align Before Aggregation: Basis-Consistent Federated LoRA under Heterogeneous Ranks

Pengpeng Qiao, Yang Cao, Lingling Zhang, Guo Cheng, Junwei Chen, Manjiang Yu, Wei Yang Bryan Lim, Masatoshi Yoshikawa

NeurIPS 2026 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated prompt and LoRA tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a basis-consistent server-side aggregation for federated LoRA with heterogeneous client ranks, aligning LoRA factor bases before aggregation rather than composing per-client matrix updates directly.

Problem

Federated fine-tuning of LLMs with LoRA when clients have heterogeneous ranks; existing server aggregators compose local updates ΔW_i from client-specific (A_i, B_i) factors before aggregating, which the abstract suggests is problematic.

Approach

Instead of aggregating composed matrix updates, align the LoRA factor bases of heterogeneous-rank clients before aggregation so that the aggregated update is basis-consistent. Details beyond the abstract are not available.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add basis-consistent reconstruction under the global reference basis (ablation)FedAbA · NLU avg, high heterogeneityaverage score: 82.02 -> 82.90Improved
replace reconstruction with factor rotation only (ablation)FedAbA · NLU avg, high heterogeneityaverage score: 82.07 vs 82.02 withoutNo clear effect
use orthogonal Procrustes alignment instead of generic orthonormalization (ablation)FedAbA · NLU avg, high heterogeneityaverage score: 82.32 -> 82.90Improved
add consistency-score weighting c_i to data-size weights (ablation)FedAbA · NLG MMLU, high heterogeneityaccuracy: 40.40 -> 40.90Improved
add data-size term alongside consistency weight (ablation)FedAbA · NLG MMLU, high heterogeneityaccuracy: 40.55 -> 40.90Improved
add gauge fixing before rank-prefix dispatch (ablation)FedAbA · NLU avg, high heterogeneityaverage score: 82.52 -> 82.90Improved
increase rank heterogeneity from homogeneous to highFedAbA vs Fed-PLoRA · NLU avgaverage score gap: 80.97 vs 80.75 -> 82.90 vs 82.10Improved
align before aggregation on low-rank clientsFedAbA vs Fed-PLoRA · Dolly-15K r=4 clientsROUGE-L: 29.7 -> 31.2Improved
replace dense full-update SVD with coefficient-space SVDFedAbA vs FlexLoRA · server costaggregation time: 130s -> 2.0sImproved
use FedAbA instead of Fed-PLoRA on larger backbonesFedAbA · Qwen3-4B and LLaMA-2-13B Dolly-15KMMLU: +0.7 pointsImproved
add a short warm-start (ablation)FedAbA · high heterogeneity sensitivityperformanceImproved

Experimental setup

Builds on: LoRA

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy