Research map / Continual learning / Continual fine-tuning of LLMs
Closed-Form Merging of Parameter-Efficient Modules for Federated Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRM, a closed-form, alternating-optimization method for merging LoRA modules via RegMean-style regression, applied to federated class-incremental learning.
Problem
Merging LoRA modules across clients and tasks in federated class-incremental learning, where the closed-form merge of LoRA's A and B matrices is indeterminate when both are unknown.
Approach
Fix one LoRA matrix (A or B) shared across clients and solve the other in closed form via RegMean; alternate which matrix is updated each communication round, and apply RegMean to full task residuals across tasks.
Key results
- LoRM achieves FAA 86.95 on CIFAR-100 (beta=0.5) vs 83.88 for the next best (L2P), and 72.48 on ImageNet-R (beta=0.5) vs 70.00 for CCVR.
- On EuroSAT (beta=1.0) LoRM reaches 84.23 FAA vs 73.38 for CODA-Prompt.
- On Out-of-Scope, LoRM reaches 84.78 FAA (beta=1.0) vs 74.16 for RegMean.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoRM instead of CCVR | FCIL on ImageNet-R (β=0.5) · ImageNet-R (β=0.5) | FAA: 70.00 -> 72.48 | Helped |
| Use LoRM instead of CCVR | FCIL on ImageNet-A (β=0.5) · ImageNet-A (β=0.5) | FAA: 36.27 -> 36.34 | No clear effect |
| Use LoRM instead of CODA-P | FCIL on EuroSAT (β=1.0) · EuroSAT (β=1.0) | FAA: 73.38 -> 84.23 | Helped |
| Use LoRM instead of PILoRA | FCIL on CUB-200 (β=0.2) · CUB-200 (β=0.2) | FAA: 60.39 -> 60.06 | No clear effect |
| Add LoRA to FedAvg (ablation) | FedAvg on EuroSAT (β=1.0) · EuroSAT (β=1.0) | FAA: 41.04 -> 53.14 | Helped |
| Add LoRA to FedAvg (ablation) | FedAvg on ImageNet-R (β=0.5) · ImageNet-R (β=0.5) | FAA: 57.65 -> 57.00 | No clear effect |
| Add alternating optimization and omit Eq. [ref] (average task-specific residuals) (ablation) | RegMean baseline on ImageNet-R (β=0.5) · ImageNet-R (β=0.5) | FAA: 61.18 -> 75.08 | Helped |
| Add alternating optimization and omit Eq. [ref] (average task-specific residuals) (ablation) | RegMean baseline on EuroSAT (β=1.0) · EuroSAT (β=1.0) | FAA: 48.74 -> 71.40 | Helped |
| Add Eq. [ref] to merge task-specific residuals (ablation) | LoRM w/o Eq. on ImageNet-R (β=0.5) · ImageNet-R (β=0.5) | FAA: 75.08 -> 75.62 | Helped |
| Add Eq. [ref] to merge task-specific residuals (ablation) | LoRM w/o Eq. on EuroSAT (β=1.0) · EuroSAT (β=1.0) | FAA: 71.40 -> 77.37 | Helped |
| Alternate training between matrices A and B rather than train only B (ablation) | LoRM on EuroSAT (β=1.0) · EuroSAT (β=1.0) | FAA | Helped |
Surprising findings
- Closed-form merging of A and B jointly is indeterminate; fixing one matrix per round resolves it.
- CCVR, a simple server-side calibration baseline, outperforms the FCIL-specific TARGET and PILoRA on ImageNet-R.
Limitations
- Closed-form merging requires fixing one of the two LoRA matrices per round, a workaround rather than a joint solution.
- Evaluated only on FCIL and a single LoRA-based setup with ViT-B/16 and T5-small backbones.
Experimental setup
Datasets: CIFAR-100 ImageNet-R ImageNet-A Cars-196 CUB-200 EuroSAT Out-Of-Scope
Benchmarks: CIFAR-100 ImageNet-R ImageNet-A Cars-196 CUB-200 EuroSAT Out-Of-Scope
Metrics: Final Average Accuracy (FAA)
Baselines: EWC LwF L2P CODA-Prompt FisherAvg RegMean CCVR FedProto TARGET PILoRA TIES-Merging
Builds on: RegMean LoRA PILoRA TARGET ACE
Closest papers by meaning
- Selective Aggregation for Low-Rank Adaptation in Federated Learning (ICLR 2025 · similarity 0.87)
- When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation (NeurIPS 2026 · similarity 0.87)
- Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA (NeurIPS 2025 · similarity 0.87)
- IterIS: Iterative Inference-Solving Alignment for LoRA Merging (CVPR 2025 · similarity 0.87)
- Accurate and Efficient Low-Rank Model Merging in Core Space (NeurIPS 2025 · similarity 0.87)
- SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.