Research map / Large language models / Instruction and fine-tuning
SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeedLoRA, a two-stage fusion of LoRA adapters trained on the same task with different random seeds (robust/conflict dimension handling plus SVD subspace fusion), narrowing the gap to full fine-tuning on math and code.
Problem
LoRA often lags full fine-tuning on complex tasks like mathematical reasoning and code generation, and existing model merging methods are designed for multi-task rather than single-task settings.
Approach
Merge LoRA adapters trained on the same task with different seeds: average robust same-sign large entries, resolve conflicting-sign entries by majority sign, and fuse residual entries via truncated SVD subspace projection and TIES/DARE/averaging in the shared low-rank basis.
Key results
- On LLaMA2-7B with MetaMathQA, SeedLoRA raises LoRA (r=8) GSM8K from 64.1 to 68.9 (full FT 66.5) and MATH from 15.3 to 17.8.
- On Code-Feedback with LLaMA2-7B, SeedLoRA improves HumanEval from 34.1 to 40.7 (full FT 40.3).
- On LLaMA2-13B, SeedLoRA gives about 2.3% gain over the best individual LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Merge LoRA models trained with different seeds | LLaMA2-7B LoRA (r=8) · GSM8K | accuracy: 64.1 to 68.9 | Helped |
| Merge LoRA models trained with different seeds | LLaMA2-7B LoRA (r=8) · MATH | accuracy: 15.3 to 17.8 | Helped |
| Merge LoRA models trained with different seeds | Mistral-7B LoRA (r=8) · GSM8K | accuracy | Helped |
| Merge LoRA models trained with different seeds | LLaMA2-7B LoRA · HumanEval | pass rate: 34.1 to 40.7 | Helped |
| Merge LoRA models trained with different seeds | LLaMA-2-7B LoRA · GSM8K | accuracy: 22.5 to 22.0 | Mixed |
| Merge LoRA models trained with different seeds | LLaMA-2-7B LoRA · Average | average score: 36.4 to 37.3 | Helped |
| Merge LoRA+ models trained with different seeds | LLaMA-2-7B LoRA+ · Average | average score: 38.1 to 38.8 | Helped |
| Merge DoRA models trained with different seeds | LLaMA-2-7B DoRA · Average | average score: 37.5 to 38.5 | Helped |
| Use a single LoRA model with rank 24 instead of merging three rank-8 models (ablation) | LLaMA2-7B · MetaMathQA and Code-Feedback | performance | Hurt |
| Train three LoRA models for one epoch each and merge them rather than train one model for three epochs (ablation) | LLaMA2-7B LoRA · MetaMathQA | performance | Helped |
| Train vanilla LoRA for more epochs instead of applying SeedLoRA (ablation) | LLaMA2-7B LoRA · MetaMathQA and Code-Feedback | performance | Hurt |
Surprising findings
- SeedLoRA outperforms a single LoRA with 3x rank (r=24) and LoRA trained for more epochs.
- Training three 1-epoch LoRAs and merging matched or beat standard 3-epoch LoRA training at the same cost.
Limitations
- Gains on general instruction-following (Tulu-v2) are smaller than on math and code.
- Requires training or obtaining several LoRA adapters on the same task, adding upfront cost.
- Many results are reported with few seeds (3) and small absolute differences.
Experimental setup
Datasets: MetaMathQA Code-Feedback Tulu-v2
Benchmarks: GSM8K MATH HumanEval HumanEval+ MMLU BBH TyDiQA TruthfulQA
Metrics: accuracy pass@1
Baselines: Model Soup TIES DARE LoRA+ DoRA Full fine-tuning Higher-rank LoRA (r=24)
Builds on: LoRA TIES-Merging DARE Model Soup
Closest papers by meaning
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.93)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.92)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.91)
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.90)
- LoRA vs Full Fine-tuning: An Illusion of Equivalence (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.