Research map / Architectures and efficiency / Quantization
Training-Free Activation Sparsity in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TEAL, a training-free method that applies magnitude-based activation sparsity to every matrix input in LLaMA-style models, with a greedy per-layer sparsity allocation and a custom sparse GEMV kernel for decoding speed-ups.
Problem
Activation sparsity for speeding up memory-bound LLM decoding has relied on ReLU-based models or costly continued pre-training; modern SwiGLU LLMs have dense activations, and prior training-free methods (CATS) sparsify only part of the model.
Approach
Zero out low-magnitude entries of the input to every linear layer (input sparsity) using per-tensor thresholds estimated offline from generic text, with block-wise greedy allocation of sparsity across the seven matrices of each block, so that 40-50% model-wide sparsity is achieved training-free.
Key results
- TEAL at 40% model-wide sparsity keeps Llama-3-8B WikiText perplexity at 6.21 (baseline 5.87) while CATS at 40% degrades to 7.6e4.
- Single-batch decoding speed-ups of up to 1.53x at 40% and 1.8x at 50% sparsity on A6000 for Llama-2 models.
- TEAL 50% outperforms MaskLLM 2:4 weight sparsity on Llama-3-8B perplexity (6.67 vs 8.53) without any training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TEAL 25% instead of CATS 25% | Llama-3-8B · Six-task average | average score: 64.15 -> 67.73 | Helped |
| Apply TEAL 40% instead of CATS 25% | Llama-3-70B · Six-task average | average score: 79.25 -> 79.29 | Helped |
| Increase TEAL sparsity from 25% to 40% | Llama-3-8B · WikiText validation | perplexity: 5.94 -> 6.21 | Hurt |
| Increase TEAL sparsity from 25% to 40% | Llama-3-8B · Six-task average | average score: 67.73 -> 66.21 | Hurt |
| Use greedy layer-level sparsities instead of uniform sparsity (ablation) | TEAL on Llama-3-8B · Six-task average | average score: Uniform 25: 67.63; Greedy 25: 67.73; Uniform 40: 65.35; Greedy 40: 66.21; Uniform 50: 62.67; Greedy 50: 63.42; Uniform 6 | Helped |
| Use input sparsity rather than output sparsity for W_up (ablation) | TEAL on SwiGLU models · W_up activation error | activation error | Helped |
| Fine-tune with LoRA rank 32 (ablation) | TEAL on Llama-3-8B · WikiText at 50% sparsity | perplexity: 6.673 -> 6.622 | Helped |
| Fine-tune with LoRA rank 32 (ablation) | TEAL on Llama-3-8B · WikiText at 70% sparsity | perplexity: 13.39 -> 9.927 | Helped |
| Fine-tune with LoRA rank 32 (ablation) | TEAL on Llama-3-8B · WikiText at 90% sparsity | perplexity: 4.141 · 10⁵ -> 4.589 · 10³ | Helped |
| Combine TEAL with MaskLLM 2:4 weight sparsity (ablation) | Llama-3-8B · WikiText | perplexity: 6.673 -> 9.590 | Hurt |
| Sparsify the initial prefill tokens (ablation) | TEAL · Prefill | perplexity | Hurt |
| Increase activation sparsity | TEAL on Llama-2-7B · Single-batch decoding on A6000 | speed-up: 1.78× | Helped |
Surprising findings
- Input sparsity on W_up outperforms the output sparsity used by CATS, contrary to the intuition that the SiLU gate should serve as the mask.
- Llama-3 models degrade more than Llama-2/Mistral at 50% sparsity, which the authors link to quantization being less effective on models trained on more tokens.
Limitations
- Speed-ups diminish at higher batch sizes since activation sparsity relies on memory-bound low-batch inference.
- Does not sparsify the LM head, limiting speed-ups for Llama-3-8B.
- Prefill must be sparsified for log-likelihood tasks, and initial attention-sink tokens must be protected.
Experimental setup
Datasets: C4 WikiText MMLU ARC HellaSwag GSM8K PiQA WinoGrande
Benchmarks: WikiText MMLU ARC challenge HellaSwag GSM8K PiQA WinoGrande
Metrics: perplexity accuracy average accuracy tokens per second speed-up
Baselines: CATS Deja Vu kernel MaskLLM ReLUfication
Builds on: CATS DejaVu GPT-Fast AWQ
Closest papers by meaning
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.92)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.92)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.92)
- Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.91)
- ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models (ICLR 2024 · similarity 0.90)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.