Research map / Architectures and efficiency / Pruning and efficient tuning
R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R-Sparse, a training-free method that sparsifies input activation channels and approximates the remaining computation with a low-rank SVD path, with an evolutionary search for per-layer sparse/rank ratios, reaching 50% model-level sparsity on Llama-2/3 and Mistral.
Problem
Activation sparsity for LLM inference is hard to apply training-free to non-ReLU LLMs (SiLU/GELU), suffers from difficulty predicting active output channels, and reaches limited model-level sparsity.
Approach
Decompose each linear layer into an input-activation sparse part (keep high-magnitude input channels) and a low-rank SVD part applied to the dropped channels, leveraging that important contributions concentrate in the bottom-right of the channel-by-singular-value contribution matrix; per-layer sparse/rank ratios are found by evolutionary search over perplexity.
Key results
- At 50% model-level sparsity, R-Sparse averages 64.06 on Llama-2-7B vs 65.88 full, and 66.20 on Llama-3-8B vs 69.44 full, versus GRIFFIN_50% at 45.91 on Llama-2-7B.
- R-Sparse_40% on Llama-2-7B reaches 65.00 average, outperforming CATS_40% (46.26) by about 18.7 points.
- End-to-end generation speed improves by up to 42% on Llama-2-7B and 40% on Llama-3-8B with a customized Triton kernel on a single A6000 (FP32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply R-Sparse at 50% model-level sparsity instead of GRIFFIN at 50% | Llama-2-7B · Common-sense reasoning (8-task average) | accuracy: 45.91 -> 64.06 | Helped |
| Apply R-Sparse at 50% model-level sparsity instead of GRIFFIN at 50% | Llama-3-8B · Common-sense reasoning (8-task average) | accuracy: 45.11 -> 66.20 | Helped |
| Apply R-Sparse at 50% model-level sparsity instead of GRIFFIN at 50% | Mistral-7B · Common-sense reasoning (8-task average) | accuracy: 48.07 -> 68.39 | Helped |
| Combine sparse activation computation with a low-rank component rather than using vanilla activation sparsity alone (ablation) | R-Sparse on Llama-2-7B · Common-sense reasoning (WG, PIQA, SciQ, OBQA average) | accuracy: 66.25 -> 67.50 | Helped |
| Add a sparse component to low-rank decomposition (ablation) | R-Sparse on Llama-2-7B · Common-sense reasoning (WG, PIQA, SciQ, OBQA average) | accuracy: 33.05 -> 67.50 | Helped |
| Combine R-Sparse with INT4 weight quantization at 40% sparsity | Llama-2-7B · Common-sense reasoning (WG, PIQA, SciQ, OBQA average) | accuracy: 67.32 -> 66.41 | Hurt |
| Combine R-Sparse with INT4 weight quantization at 50% sparsity | Llama-2-7B · Common-sense reasoning (WG, PIQA, SciQ, OBQA average) | accuracy: 67.32 -> 65.76 | Hurt |
| Use an adaptive searched sparsification recipe instead of a uniform recipe (ablation) | R-Sparse · OBQA | accuracy: +1.95 | Helped |
| Use an adaptive searched sparsification recipe instead of a uniform recipe (ablation) | R-Sparse · ARC-E | accuracy: +0.89 | Helped |
| Use an adaptive searched sparsification recipe instead of a uniform recipe (ablation) | R-Sparse · ARC-C | accuracy: +1.13 | Helped |
| Use an adaptive searched sparsification recipe instead of a uniform recipe (ablation) | R-Sparse · BoolQ | accuracy: +1.11 | Helped |
| Use a customized Triton kernel for R-Sparse inference | Llama-2-7B · Generation speed | generation speed improvement: 42% | Helped |
Surprising findings
- Simply replacing SiLU with ReLU (ReLUfication) without training collapses performance (e.g., 35.34 average on Llama-2-7B).
- Pure low-rank decomposition at the same budget fails badly (33.05 average vs 68.10 full on Llama-2-7B subset), while pure sparsity reaches 66.25.
Limitations
- Evaluated only at decoding stage with simulated prefill; gains are measured on simulated prompts and a single GPU.
- Adaptive sparsity recipe is searched on a small set (16 C4 samples) and the search is specific to each model.
- Performance degrades noticeably at 60-70% sparsity on some tasks (e.g., OBQA drops to 24.0 at 70% adaptive).
Experimental setup
Datasets: C4
Benchmarks: Winogrande PIQA SciQ OpenBookQA HellaSwag BoolQ ARC-Easy ARC-Challenge XSUM WikiText-2
Metrics: accuracy Rouge-L perplexity
Baselines: ReLUfication CATS GRIFFIN vanilla activation sparsity low-rank decomposition uniform sparsification recipe
Builds on: CATS GRIFFIN ReLUfication GPTQ
Closest papers by meaning
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.94)
- Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.93)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.92)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.92)
- Training-Free Activation Sparsity in Large Language Models (ICLR 2025 · similarity 0.92)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.