Research map / Architectures and efficiency / Pruning and efficient tuning
SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SparseLoRA, which accelerates LoRA fine-tuning by applying training-free, input-dependent structured channel sparsity to the frozen main branch, selected by a low-rank SVD estimator, with layer-wise sparsity, context-only sparsity, and a dense warmup phase.
Problem
LoRA-style PEFT methods reduce trainable parameters and memory but not computation, and can even slow fine-tuning; LLM fine-tuning needs compute reduction too.
Approach
Extends contextual activation sparsity from inference to fine-tuning: an offline SVD-based, training-free sparsity estimator selects input-dependent weight channels (L2-norm for FFN and VO, attention-score norm for QK) for the main branch, applied non-uniformly across layers, only to context tokens, and after a dense initial phase, while LoRA branches stay dense.
Key results
- On LLaMA3-8B Math10K, SparseLoRA reaches 1.6x speedup at 46% FLOPs with 81.1 average accuracy vs LoRA 81.0.
- On LLaMA2-13B CSR170K, SparseLoRA achieves 85.0 average accuracy vs LoRA 84.7 at 61% FLOPs and 1.3x speedup.
- The SVD sparsity estimator adds 0.05% FLOPs and 0.8% runtime overhead, with accuracy 81.1 vs oracle 81.4 on Math10K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add contextual sparsity to LoRA fine-tuning | LLaMA2-13B · CSR170K | Average accuracy: 84.7 -> 85.0 | Helped |
| Add contextual sparsity to LoRA fine-tuning | LLaMA3-8B · Math10K | Speedup: 1.0$\times$ -> 1.6$\times$ | Helped |
| Add contextual sparsity to LoRA fine-tuning | LLaMA3-8B · CSR170K | Average accuracy: 87.1 -> 86.9 | Hurt |
| Add contextual sparsity to LoRA fine-tuning | LLaMA3-8B · Math10K | Average accuracy: 81.0 -> 81.1 | Helped |
| Add contextual sparsity to LoRA fine-tuning | LLaMA3.1-8B · MT-Bench | Average score: 6.03 -> 6.06 | Helped |
| Add contextual sparsity to LoRA fine-tuning | LLaMA3.1-8B · HumanEval | Score: 43.1 -> 43.9 | Helped |
| Add contextual sparsity to QLoRA | LLaMA3-8B · Math10K | Accuracy: 80.6 -> 80.8 | Helped |
| Preserve dense computation for output tokens and sparsify context tokens (ablation) | SparseLoRA · Math10K | Accuracy: 70.9 -> 81.1 | Helped |
| Use sensitivity-guided nonuniform layer sparsity instead of uniform sparsity (ablation) | SparseLoRA · Math10K | Accuracy: 80.3 -> 81.1; 80.2 -> 81.1; 79.5 -> 80.5 | Helped |
| Use channel-wise rather than head-wise QK selection (ablation) | SparseLoRA · Math10K | Accuracy: 79.6 -> 80.7 | Helped |
| Use attention norm rather than L2 norm for QK pruning (ablation) | SparseLoRA · Math10K | Accuracy: 79.8 -> 80.7 | Helped |
| Use L2 norm rather than random pruning for FFN channels (ablation) | SparseLoRA · Math10K | Accuracy: 78.6 -> 81.4 | Helped |
Surprising findings
- DoRA is about 20-30% slower than LoRA in practice despite being a PEFT method, due to overhead.
- Output tokens are much more sensitive to sparsity than context tokens, so uniform sparsity over all tokens drops Math10K accuracy from about 80 to 47.1.
Limitations
- Speedups vary by dataset and model (1.2x-1.6x), and some configurations lose accuracy slightly on individual tasks.
- Sparsity configurations (layers and ratios) are tuned per model and dataset via sensitivity analysis.
- Evaluated mainly on LLaMA-family models of 7-13B parameters.
Experimental setup
Datasets: CSR170K Math10K GLUE WizardLM CodeFeedback
Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA GSM8K SVAMP MAWPS GLUE
Metrics: accuracy #FLOPs (% of LoRA) speedup runtime
Baselines: LoRA QLoRA DoRA GaLore Wanda
Builds on: LoRA DoRA Deja Vu Wanda GaLore
Closest papers by meaning
- S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.92)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.90)
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.89)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.88)
- DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs (NeurIPS 2025 · similarity 0.88)
- SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.