Research map / Federated learning and privacy / Personalized and heterogeneous FL

Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning

NeurIPS 2025 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated prompt and LoRA tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FedLEASE, which clusters federated clients by cosine similarity of LoRA B matrices (silhouette-selected number of experts) and uses an adaptive top-M router over 2M-1 outputs so each client selects a variable number of experts.

Problem

Federated LoRA fine-tuning of LLMs over heterogeneous clients (different domains/tasks) where a single shared LoRA module underperforms, and it is unclear how many LoRA experts to allocate and how many each client should use.

Approach

Allocate experts by clustering clients on cosine similarity of briefly trained LoRA B matrices, choosing the cluster count by silhouette score; then give each client a router over 2M-1 outputs (its assigned expert's M outputs plus M-1 others) with a top-M selection so the number of experts used per input adapts from one to all.

Key results

• On GLUE with RoBERTa-Large (16 clients), FedLEASE averages 87.76% accuracy vs 84.60% for the strongest baseline FedSA, a +3.16 point average gain.
• On FLAN NLG tasks with LLaMA-2-7B (8 clients), FedLEASE improves average ROUGE-1 by 1.50% over the strongest baseline.
• Silhouette analysis selects 4 clusters/experts for the 16-client GLUE setting; using Spectral clustering instead gives comparable average accuracy (87.62 vs 87.76).

Measured effects reported in the paper

Surprising findings

• FedIT outperforms FedLoRA-MoE when all clients share the same task, indicating separate experts can be redundant in homogeneous settings.
• Similarity from LoRA B matrices reflects task identity while A matrices show no consistent task-similarity pattern.

Limitations

• Assumes a static client population and fixed expert assignments throughout training.
• Clustering and expert count are chosen once at initialization, not adapted dynamically.
• Convergence analysis relies on strong convexity and smoothness assumptions that do not hold for deep LLMs.

Experimental setup

Datasets: GLUE (SST-2, QNLI, MRPC, QQP) FLAN (Text Editing, Struct to Text, Sentiment Analysis, Commonsense Reasoning) QQP (Dirichlet label non-IID)

Benchmarks: GLUE SST-2 GLUE QNLI GLUE MRPC GLUE QQP FLAN text editing FLAN struct to text FLAN sentiment analysis FLAN commonsense reasoning

Metrics: accuracy ROUGE-1

Baselines: FedIT FFA-LoRA FedDPA FedSA IFCA+LoRA

Builds on: LoRA FedAvg IFCA FedSA FedIT

Closest papers by meaning

• Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA (NeurIPS 2025 · similarity 0.93)
• FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations (NeurIPS 2024 · similarity 0.91)
• Selective Aggregation for Low-Rank Adaptation in Federated Learning (ICLR 2025 · similarity 0.91)
• Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.91)
• Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning (NeurIPS 2026 · similarity 0.90)
• Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources (NeurIPS 2024 · similarity 0.90)