Research map / Architectures and efficiency / Pruning and efficient tuning
Dynamic Low-Rank Sparse Adaptation for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoSA, which sparsifies LoRA adapters with the same mask as the sparse LLM and allocates layer-wise sparsity (via representation mutual information) and LoRA ranks (via reconstruction error) during iterative sparse fine-tuning, so adapters merge into sparse weights without added latency.
Problem
Fine-tuning unstructured-sparse LLMs with LoRA yields adapters that cannot be merged into sparse weights (adding inference latency), and uniform layer-wise sparsity and uniform LoRA ranks hurt recovery at high sparsity.
Approach
Dynamically sparsify LoRA outputs with the LLM's sparse mask so adapters merge post-training; set layer-wise sparsity from RMI approximated by normalized HSIC on feature maps; allocate LoRA ranks proportional to layer reconstruction errors; increase sparsity and rank gradually over T steps with a cubic schedule.
Key results
- At 70% sparsity, LoSA reduces the perplexity of Wanda-sparsified LLaMA-2-7B by 68.73 (79.67 to 10.94) and raises mean zero-shot accuracy by 16.32 points.
- LoSA reaches 2.60x CPU and 2.23x GPU speedup for 70% sparse LLaMA-2-7B, versus LoRA's 2.29x CPU and 1.71x GPU at the same sparsity (LoRA's unmerged adapters add latency).
- Fine-tuning a 70% sparse LLaMA-2-7B takes about 45 minutes on a single A100 80GB GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add layer-wise sparsity rates (LSR) (ablation) | LoSA on Wanda 70% sparse LLaMA-2-7B · Wanda 70% sparsity, LLaMA-2-7B | accuracy: 50.62 -> 51.65 | Helped |
| Add sparsity-aware rank allocation (SRA) (ablation) | LoSA on Wanda 70% sparse LLaMA-2-7B · Wanda 70% sparsity, LLaMA-2-7B | accuracy: 50.94 -> 51.65 | Helped |
| Add dynamic sparsity and adaptation (DSA) (ablation) | LoSA on Wanda 70% sparse LLaMA-2-7B · Wanda 70% sparsity, LLaMA-2-7B | accuracy: 49.90 -> 51.65 | Helped |
| Add LSR and SRA (ablation) | LoSA on Wanda 70% sparse LLaMA-2-7B · Wanda 70% sparsity, LLaMA-2-7B | accuracy: 48.81 -> 51.65 | Helped |
| Add LSR and DSA (ablation) | LoSA on Wanda 70% sparse LLaMA-2-7B · Wanda 70% sparsity, LLaMA-2-7B | accuracy: 48.30 -> 51.65 | Helped |
| Add SRA and DSA (ablation) | LoSA on Wanda 70% sparse LLaMA-2-7B · Wanda 70% sparsity, LLaMA-2-7B | accuracy: 48.54 -> 51.65 | Helped |
| Add LSR, SRA, and DSA (ablation) | LoSA on Wanda 70% sparse LLaMA-2-7B · Wanda 70% sparsity, LLaMA-2-7B | accuracy: 47.76 -> 51.65 | Helped |
| Allocate ranks by reconstruction error instead of sparsity rate (ablation) | LoSA on Wanda 70% sparse LLaMA-2-7B · Wanda 70% sparsity, LLaMA-2-7B | perplexity: 11.37 -> 10.94 | Helped |
| Use cubic instead of linear sparsity schedule (ablation) | LoSA on Wanda 60% sparse LLaMA-2-7B · Wanda 60% sparsity, LLaMA-2-7B | perplexity: 8.04 -> 7.88 | Helped |
| Apply low-rank adaptation to all linear layers instead of a subset (ablation) | LoSA on Wanda 60% sparse LLaMA-2-7B · Wanda 60% sparsity, LLaMA-2-7B | perplexity: 8.22 -> 7.88 | Helped |
| Use LoSA instead of LoRA | Wanda 70% sparse LLaMA-2-7B · WikiText-2, 70% sparsity | perplexity: 12.57 -> 10.94 | Helped |
| Use LoSA instead of LoRA | Wanda 70% sparse LLaMA-2-7B · Mean zero-shot accuracy, 70% sparsity | accuracy: 48.18 -> 51.65 | Helped |
Surprising findings
- Fine-tune-first-then-sparsify yields much worse accuracy at 70% sparsity (77.57 perplexity) than iterative sparse fine-tuning with LoSA.
- Ranking layers by reconstruction error outperforms ranking by sparsity rate for rank allocation.
Limitations
- Gap remains to lossless high-ratio sparsity; accuracy at 70% sparsity is still below dense.
- Fine-tuning takes longer than LoRA due to T=5 sparsification rounds and layer-wise computations.
Experimental setup
Datasets: Alpaca-GPT4 C4 WikiText-2
Benchmarks: WikiText-2 HellaSwag Winogrande BoolQ OpenBookQA PIQA ARC-Easy ARC-Challenge
Metrics: perplexity zero-shot accuracy speedup throughput
Baselines: SparseGPT Wanda LoRA AdaLoRA SoRA Sparse LoRA LLM-Pruner LoRAPrune
Builds on: LoRA SparseGPT Wanda Representation mutual information
Closest papers by meaning
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.92)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.92)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.92)
- NOLA: Compressing LoRA using Linear Combination of Random Basis (ICLR 2024 · similarity 0.92)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.91)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.