Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse High Rank Adapters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SHiRA, a sparse high-rank adapter that fine-tunes only 1-2% of base model weights with a static mask, enabling direct weight overwriting for fast adapter switching and reduced concept loss in multi-adapter fusion.
Problem
LoRA adapters, once fused, modify the entire weight tensor, hindering rapid adapter switching on mobile devices; unfused LoRA adds up to 30% inference latency, and naive multi-adapter LoRA merging causes concept loss.
Approach
Instead of adding low-rank matrices, train only a small sparse subset (1-2%) of existing base weights selected by a static mask (magnitude, gradient, SNIP, random, or structured), so the adapter is a sparse delta that can be scattered into the base weights directly; sparsity and structured masks make adapters near-orthogonal, reducing interference in multi-adapter fusion.
Key results
- On LLaMA-7B commonsense reasoning, SHiRA-SNIP averages 77.4% vs LoRA 74.7% (+2.7%) while changing only 1% of weights versus 66.72% for LoRA when fused.
- In multi-adapter fusion on BoolQ, PIQA, and Arc-Easy, SHiRA-WM-Overlap averages 74.02 vs LoRA 67.33 (+6.69%), with 4.43% drop vs 11.08% for LoRA.
- Loading SHiRA on a CPU is 4.68x faster for SDXL and 5.71x faster for LLaMA2-7B than LoRA fusion; training uses up to 16.63% less peak GPU memory than LoRA-PEFT on LLaMA2-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LoRA with SHiRA-SNIP, training top 1% of weights selected by SNIP | LLaMA-7B · Commonsense reasoning | average accuracy: 74.7 -> 77.4 | Helped |
| Replace LoRA with SHiRA-SNIP, training top 1% of weights selected by SNIP | LLaMA2-7B · Commonsense reasoning | average accuracy: 77.61 -> 79.28 | Helped |
| Use structured trainable rows or columns plus the diagonal instead of LoRA | Stable Diffusion v1.5 · Paintings, α=1 | HPSv2 score: 24.7 ±1.8 -> 31.2 ± 1.7 | Helped |
| Use structured trainable rows or columns plus the diagonal instead of LoRA | Stable Diffusion v1.5 · Bluefire, α=1 | HPSv2 score: 32.6 ± 1.9 -> 34.2 ± 1.6 | Helped |
| Replace the structured mask with a random mask (ablation) | SHiRA · Paintings, α=1 | HPSv2 score: 31.2 ± 1.7 -> 30.7 ± 1.9 | Hurt |
| Replace the structured mask with a weight-magnitude mask (ablation) | SHiRA · Paintings, α=1 | HPSv2 score: 31.2 ± 1.7 -> 29.7 ± 1.9 | Hurt |
| Replace the structured mask with a gradient-magnitude mask (ablation) | SHiRA · Paintings, α=1 | HPSv2 score: 31.2 ± 1.7 -> 30.3 ± 1.8 | Hurt |
| Replace the structured mask with a SNIP mask (ablation) | SHiRA · Paintings, α=1 | HPSv2 score: 31.2 ± 1.7 -> 29.8 ± 1.8 | Hurt |
| Replace the structured mask with a random mask (ablation) | SHiRA · Bluefire, α=1 | HPSv2 score: 34.2 ± 1.6 -> 33.4 ± 1.9 | Hurt |
| Replace the structured mask with a weight-magnitude mask (ablation) | SHiRA · Bluefire, α=1 | HPSv2 score: 34.2 ± 1.6 -> 31.9 ± 2.1 | Hurt |
| Replace the structured mask with a gradient-magnitude mask (ablation) | SHiRA · Bluefire, α=1 | HPSv2 score: 34.2 ± 1.6; 34.2 ± 1.5 | No clear effect |
| Replace the structured mask with a SNIP mask (ablation) | SHiRA · Bluefire, α=1 | HPSv2 score: 34.2 ± 1.6 -> 33.7 ± 1.7 | Hurt |
Surprising findings
- SHiRA-Struct, the best masking method for vision, performs poorly on language tasks because it is effectively rank-1 plus diagonal.
- Non-overlapping sparse masks reduce single-adapter accuracy relative to overlapping masks but improve multi-adapter fusion.
Limitations
- SHiRA-Struct underperforms on language tasks due to its rank-1 plus diagonal structure, so the best mask depends on the domain.
- Orthogonality analysis covers adapter weights only, not subspace orthogonality, which is left for future work.
- Non-overlapping masks trade off single-adapter accuracy for multi-adapter fusion, and optimal mask design is left open.
Experimental setup
Datasets: BoolQ PIQA SIQA OBQA Winogrande HellaSwag ARC-easy ARC-challenge Bluefire Paintings DreamBooth dataset cifar10
Benchmarks: BoolQ PIQA SIQA OBQA Winogrande HellaSwag ARC-easy ARC-challenge HPSv2
Metrics: accuracy HPSv2 score average accuracy drop in multi-adapter fusion switching latency peak GPU memory AWOM/AWOR orthogonality
Baselines: LoRA DoRA LoRA+ SpIEL SoRA RoSA
Builds on: LoRA DoRA SNIP Sparse-Adapters DreamBooth
Closest papers by meaning
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.91)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.90)
- $\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning (NeurIPS 2024 · similarity 0.90)
- LoRA+: Efficient Low Rank Adaptation of Large Models (ICML 2024 · similarity 0.90)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.