Research map / Large language models / Instruction and fine-tuning
LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRASuite, a training-light method that transfers existing LoRA weights across LLM upgrades via embedding-based transfer matrices, CKA-based layer mapping, Hungarian head mapping, and a short fine-tuning step.
Problem
LoRA weights trained on an earlier LLM version become unusable after the backbone is upgraded (changes in vocabulary, hidden size, intermediate size, depth, head count, attention type), and retraining them from scratch is costly and time-consuming.
Approach
Map LoRA weights from old to new LLM via a transfer matrix from embeddings, layer alignment using minibatch CKA with dynamic programming, and attention-head alignment using input-independent QK/VO interaction matrices with cosine similarity and the Hungarian algorithm, followed by a single SVD to recover low-rank factors and a small fine-tuning step.
Key results
- On MiniCPM-S-1B to MiniCPM-2B math tasks, LoRASuite with LFT (100 samples) averages 43.80 vs 42.39 for full LoRA (10k) and 22.82 for small LoRA (100).
- Memory is reduced by 5.5 GB and computation time by 78.23% relative to LoRA retraining, per the paper.
- Upgrading Qwen-1.5-1.8B to Qwen-2.5-3B on math, LoRASuite w LFT (100) averages 56.88 vs 50.32 for full LoRA (10k).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transfer old LoRA weights and add lightweight fine-tuning on 100 examples instead of retraining on 10k examples | MiniCPM-S-1B to MiniCPM-2B · Math tasks | Avg.: 42.39 -> 43.80 | Helped |
| Transfer old LoRA weights and add lightweight fine-tuning on 100 examples instead of same-scale LoRA training | MiniCPM-S-1B to MiniCPM-2B · Math tasks | Avg.: 22.82 -> 43.80 | Helped |
| Omit lightweight fine-tuning (ablation) | LoRASuite on MiniCPM-S-1B to MiniCPM-2B · Math tasks | Avg.: 23.96 -> 43.80 | Helped |
| Omit lightweight fine-tuning (ablation) | LoRASuite on MiniCPM-S-1B to MiniCPM-2B · Commonsense tasks | Avg.: 32.69 -> 42.34 | Helped |
| Transfer old LoRA weights and add lightweight fine-tuning on 100 examples instead of full-data retraining | Qwen-1.5-1.8B to Qwen-2.5-3B · Math tasks | Avg.: 50.32 -> 56.88 | Helped |
| Transfer old LoRA weights and add lightweight fine-tuning on 100 examples instead of full-data retraining | Qwen-1.5-1.8B to Qwen-2.5-3B · Commonsense tasks | Avg.: 56.30 -> 54.41 | Hurt |
| Use CKA-based layer mapping instead of sequential, middle-out, or reverse-order mapping (ablation) | LoRASuite on MiniCPM-S-1B to MiniCPM-2B · Math and commonsense tasks | Performance | Helped |
| Use minibatch CKA instead of CCA, PWCCA, or Procrustes similarity (ablation) | LoRASuite on MiniCPM-S-1B to MiniCPM-2B · Math and commonsense tasks | Performance | Helped |
| Use Hungarian-based similarity matching instead of default head ordering (ablation) | LoRASuite on MiniCPM-S-1B to MiniCPM-2B · Math and commonsense tasks | Performance | Helped |
| Increase the LFT learning rate from 1e-4 to 9e-4 (ablation) | LoRASuite on MiniCPM-S-1B to MiniCPM-2B · Math tasks | Average performance over LoRA (Small): 0.27 percentage points -> 21.30 points | Helped |
| Increase the LFT dataset size (ablation) | LoRASuite · Math tasks | Performance | Hurt |
| Apply LoRASuite rather than small-scale vanilla LoRA | MiniCPM-S-1B to MiniCPM-2B · Memory and computational time | Memory reduction: 5.5 GB | Helped |
Surprising findings
- LoRASuite without fine-tuning performs nearly identical to the vanilla upgraded model on math and commonsense tasks, indicating matrix-multiplication transfer alone is numerically unstable.
- Performance gains from LoRASuite are sensitive to the fine-tuning learning rate, with a 9e-4 rate yielding ~21 points over small LoRA while 1e-4 yields only 0.27.
Limitations
- Still requires a small-scale fine-tuning step to reach best performance.
- Only addresses explicit architectural upgrades within the same architecture family; implicit upgrades (pretraining data, RLHF) are not handled.
- Sensitive to fine-tuning learning rate and overfits with larger LFT datasets; AdaLoRA gains are smaller due to rank incompatibility.
Experimental setup
Datasets: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA AQuA GSM8K MAWPS SVAMP
Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA AQuA GSM8K MAWPS SVAMP
Metrics: accuracy average accuracy memory consumption training time
Baselines: LoRA (small-scale, 100 samples) LoRA (full, 10k samples) LoRASuite w/o LFT AdaLoRA DoRA CCA PWCCA Procrustes
Builds on: LoRA centered kernel alignment Hungarian algorithm DoRA AdaLoRA
Closest papers by meaning
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.92)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.92)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.92)
- Text-to-LoRA: Instant Transformer Adaption (ICML 2025 · similarity 0.91)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.