Research map / Federated learning and privacy / Personalized and heterogeneous FL

Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning

NeurIPS 2026 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated prompt and LoRA tuning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Fed-SB, a federated LoRA method that freezes the adapters A and B (initialized via LoRA-SB) and communicates and averages only the small r×r matrix R, giving exact aggregation with communication independent of client count and reduced DP noise.

Problem

Federated LoRA fine-tuning suffers from inexact aggregation of low-rank adapters; existing fixes either blow up communication (scaling with client count) or sacrifice expressivity, and DP noise amplifies these issues.

Approach

Build federated fine-tuning on LoRA-SB: the server distributes fixed orthonormal adapters B and A, clients train only an r×r matrix R, and averaging R gives exactly the average of client updates B R A, so communication is r² per round regardless of client count and DP noise is applied to fewer parameters without cross-term noise amplification.

Key results

• Fed-SB (r=200) on Llama-3.2 3B commonsense reasoning reaches 78.58 average accuracy vs 76.49 for FedEx-LoRA/FLoRA, while communicating 7.85M vs 243.15M parameters.
• On Mistral-7B GSM8K, Fed-SB (r=200) achieves 56.18 vs 54.28 for FedEx-LoRA/FLoRA with about 234x lower communication.
• In federated private BERT-base on SNLI at epsilon=1, Fed-SB (r=64) outperforms FedIT by 24.26 points and FFA-LoRA by 2.72 points while communicating roughly 12x and 6x fewer parameters.

Surprising findings

• FedIT degrades sharply in the federated private setting relative to the non-private setting, which the authors attribute to increased update deviation under DP noise.

Limitations

• Experiments use only 3-25 clients and a single round in most settings, so scaling advantages with many clients are asserted rather than measured.
• Fed-SB depends on fixed adapters A and B from an initialization step, and its expressivity is bounded by the chosen rank and initialization.
• Privacy results rely on a lemma with heuristic scaling arguments rather than a tight accounting analysis.

Experimental setup

Datasets: CommonSense170K MetaMathQA SNLI

Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA GSM8K MATH SNLI

Metrics: accuracy communicated parameters per round

Baselines: FedIT FedEx-LoRA FLoRA FFA-LoRA LoRA

Builds on: LoRA LoRA-SB FedAvg FedEx-LoRA FLoRA FFA-LoRA DP-SGD

Closest papers by meaning

• Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA (NeurIPS 2025 · similarity 0.93)
• Improving LoRA in Privacy-preserving Federated Learning (ICLR 2024 · similarity 0.90)
• Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.90)
• Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning (NeurIPS 2025 · similarity 0.90)
• Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix (NeurIPS 2025 · similarity 0.90)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA (NeurIPS 2025 · similarity 0.89)