Research map / Architectures and efficiency / Pruning and efficient tuning
MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixture of Shards (MoS), a LoRA variant that samples shards from global inter- and intra-layer pools via MoE-like index routing, with four differentiation tricks, achieving ~8x parameter savings over LoRA on LLaMA2-7B.
Problem
LoRA's per-customization parameter cost is high when serving many customized LLMs at once; prior parameter-sharing methods (VeRA, Tied LoRA, PRoLoRA) use only inter-layer or intra-layer sharing and pure sharing can hurt performance.
Approach
Share shards globally across layers and within layers from per-linear-type pools, select shards via index-based MoE-like routing, and add nearly cost-free differentiation (subset selection, pair dissociation, vector sharding, shard privatization) to counteract the degradation caused by excessive sharing.
Key results
- At a fixed 5.00M trainable parameters on LLaMA2-7B, MoS averages 36.39 vs. LoRA (rank 2) 34.98 and PRoLoRA 36.03.
- MoS with 19.99M parameters (rank 16/32) averages 37.63, comparable to LoRA rank 64 (37.53, 159.91M), giving about 8x parameter savings.
- On LLaMA2-13B with LoRA-equivalent rank 2 budget, MoS averages 45.98 vs. PRoLoRA 45.04 and LoRA 43.92.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add random scaling to pure sharing | Pure Sharing on LLaMA2-7B · Instruction-following tasks | Avg.: 34.33 -> 34.77 | Helped |
| add subset selection to pure sharing | Pure Sharing on LLaMA2-7B · Instruction-following tasks | Avg.: 34.33 -> 36.12 | Helped |
| add random scaling to pure sharing | Pure Sharing on LLaMA3.2-3B · Instruction-following tasks | Avg.: 43.23 -> 43.45 | Helped |
| add subset selection to pure sharing | Pure Sharing on LLaMA3.2-3B · Instruction-following tasks | Avg.: 43.23 -> 44.06 | Helped |
| remove pair dissociation (ablation) | MoS on LLaMA2-7B · Instruction-following tasks | Avg.: 37.63 -> 36.54 | Hurt |
| remove vector sharding (ablation) | MoS on LLaMA2-7B · Instruction-following tasks | Avg.: 37.63 -> 37.22 | Hurt |
| remove shard privatization (ablation) | MoS on LLaMA2-7B · Instruction-following tasks | Avg.: 37.63 -> 36.54 | Hurt |
| use independent index matrices for A and B instead of the same index matrix (ablation) | MoS on LLaMA2-7B · Instruction-following tasks | Avg.: 36.54 -> 37.63 | Helped |
| use vector shards rather than full row or column vectors (ablation) | MoS on LLaMA2-7B · Instruction-following tasks | Avg.: 0.41% | Helped |
| privatize shards for exclusive access by a low-rank matrix (ablation) | MoS on LLaMA2-7B · Instruction-following tasks | Avg.: exceeding one percent | Helped |
| combine inter-layer and intra-layer sharing with MoS differentiation | LoRA rank 2 on LLaMA2-7B · Instruction-following tasks | Avg.: 34.98 -> 36.39 | Helped |
| use MoS rather than PRoLoRA at the same parameter budget | LLaMA2-13B · Instruction-following tasks | Avg.: 45.04 -> 45.98 | Helped |
Surprising findings
- Pure parameter sharing with a much higher rank does not improve over LoRA and can underperform it on several tasks.
- Subset selection, a differentiation strategy, reverses the degradation of pure sharing and surpasses LoRA.
Limitations
- MoS requires slightly more finetuning GPU consumption and time (about 2.8% more time) than LoRA at the same rank.
- Index-based routing was chosen to avoid activation-based routing latency, but routing overhead at inference is still a concern.
- Experiments are limited to instruction-following tasks on LLaMA-family models of 3B-13B size; VeRA could not be matched in rank due to OOM.
Experimental setup
Datasets: Super-Natural Instructions Flan V2 CoT CodeAlpaca
Benchmarks: MMLU BBH GSM8K TyDi QA HumanEval
Metrics: exact match F1 pass@1 average score
Baselines: LoRA VeRA Tied LoRA PRoLoRA
Builds on: LoRA VeRA Tied LoRA PRoLoRA QLoRA
Closest papers by meaning
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.89)
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.88)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.88)
- RaSA: Rank-Sharing Low-Rank Adaptation (ICLR 2025 · similarity 0.87)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.87)
- Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.