Research map / Federated learning and privacy / Personalized and heterogeneous FL
Align Before Aggregation: Basis-Consistent Federated LoRA under Heterogeneous Ranks
Pengpeng Qiao, Yang Cao, Lingling Zhang, Guo Cheng, Junwei Chen, Manjiang Yu, Wei Yang Bryan Lim, Masatoshi Yoshikawa
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a basis-consistent server-side aggregation for federated LoRA with heterogeneous client ranks, aligning LoRA factor bases before aggregation rather than composing per-client matrix updates directly.
Problem
Federated fine-tuning of LLMs with LoRA when clients have heterogeneous ranks; existing server aggregators compose local updates ΔW_i from client-specific (A_i, B_i) factors before aggregating, which the abstract suggests is problematic.
Approach
Instead of aggregating composed matrix updates, align the LoRA factor bases of heterogeneous-rank clients before aggregation so that the aggregated update is basis-consistent. Details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add basis-consistent reconstruction under the global reference basis (ablation) | FedAbA · NLU avg, high heterogeneity | average score: 82.02 -> 82.90 | Improved |
| replace reconstruction with factor rotation only (ablation) | FedAbA · NLU avg, high heterogeneity | average score: 82.07 vs 82.02 without | No clear effect |
| use orthogonal Procrustes alignment instead of generic orthonormalization (ablation) | FedAbA · NLU avg, high heterogeneity | average score: 82.32 -> 82.90 | Improved |
| add consistency-score weighting c_i to data-size weights (ablation) | FedAbA · NLG MMLU, high heterogeneity | accuracy: 40.40 -> 40.90 | Improved |
| add data-size term alongside consistency weight (ablation) | FedAbA · NLG MMLU, high heterogeneity | accuracy: 40.55 -> 40.90 | Improved |
| add gauge fixing before rank-prefix dispatch (ablation) | FedAbA · NLU avg, high heterogeneity | average score: 82.52 -> 82.90 | Improved |
| increase rank heterogeneity from homogeneous to high | FedAbA vs Fed-PLoRA · NLU avg | average score gap: 80.97 vs 80.75 -> 82.90 vs 82.10 | Improved |
| align before aggregation on low-rank clients | FedAbA vs Fed-PLoRA · Dolly-15K r=4 clients | ROUGE-L: 29.7 -> 31.2 | Improved |
| replace dense full-update SVD with coefficient-space SVD | FedAbA vs FlexLoRA · server cost | aggregation time: 130s -> 2.0s | Improved |
| use FedAbA instead of Fed-PLoRA on larger backbones | FedAbA · Qwen3-4B and LLaMA-2-13B Dolly-15K | MMLU: +0.7 points | Improved |
| add a short warm-start (ablation) | FedAbA · high heterogeneity sensitivity | performance | Improved |
Experimental setup
Builds on: LoRA
Closest papers by meaning
- Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.90)
- Selective Aggregation for Low-Rank Adaptation in Federated Learning (ICLR 2025 · similarity 0.90)
- Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA (NeurIPS 2025 · similarity 0.89)
- Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning (NeurIPS 2026 · similarity 0.89)
- FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.88)
- Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
