Research map / Large language models / Language, safety and interpretability
Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CETT-PPL-1%, a performance-friendly metric for measuring LLM activation sparsity, and uses it to study how data amount, activation function, width-depth ratio, and scale affect sparsity, then trains a 2.4B ReLU model with 93.52% sparsity and 4.1x decoding speedup.
Problem
Activation sparsity in LLM feed-forward networks is useful for acceleration and interpretability, but there is little quantitative work on how to measure it accurately and what factors influence it.
Approach
Measure weakly-contributed neurons with CETT, tuning its hyperparameter by binary search so validation PPL rises by only 1%, and fit scaling laws of activation ratio versus pre-training data (logspace power-law for ReLU, power-law for SiLU), then apply the findings to train a sparse 2.4B ReLU model.
Key results
- CETT-PPL-1% yields a performance-neutral sparsity measure: average commonsense score change -0.16 and reading comprehension change -0.30 vs dense for 0.1B-1.2B models.
- The 2.4B ReLU model reaches a limit activation ratio of 6.48% (93.52% sparsity), fitted with a logspace power-law over ~800B tokens.
- PowerInfer decodes at 41.79 tokens/s versus 10.23 tokens/s for llama.cpp, a 4.1x speedup on one A800 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReLU instead of SiLU | LLMs with the same parameter scale · Activation sparsity measured with CETT-PPL-1% | sparsity ratio | Helped |
| Increase pre-training data for ReLU models | ReLU-activated LLMs · Pre-training | sparsity ratio | Helped |
| Increase pre-training data for SiLU models | SiLU-activated LLMs · Pre-training | sparsity ratio | Hurt |
| Use a smaller width-depth ratio below the bottleneck | 0.1B ReLU model · Width-depth ratio below about 114 | activation ratio | Helped |
| Make the width-depth ratio extremely small | Deep Transformers · Training stability and downstream performance | performance | Hurt |
| Increase parameter scale while keeping similar width-depth ratios | LLMs from 0.1B to 1.2B · Limit activation ratio | activation ratio: 2.7 points; 1.7 points | No clear effect |
| Use a smaller model | LLMs with similar width-depth ratios · Convergence toward the limit activation ratio | convergence speed | Helped |
| Use CETT-PPL-1% rather than the dense setting (ablation) | LLMs evaluated on commonsense reasoning · Commonsense reasoning average | accuracy: 49.6 -> 49.1 | No clear effect |
| Use CETT-PPL-1% rather than the dense setting (ablation) | LLMs evaluated on reading comprehension · Reading comprehension average | accuracy: 28.2 -> 28.4 | No clear effect |
| Increase CETT PPL-increase tolerance from 1% to 5% (ablation) | 0.1B ReLU model · Commonsense reasoning average | accuracy: 49.1 -> 49.2 | No clear effect |
| Increase CETT PPL-increase tolerance from 1% to 5% (ablation) | 0.1B ReLU model · Reading comprehension average | accuracy: 28.4 -> 26.9 | Hurt |
| Increase CETT PPL-increase tolerance from 1% to 10% (ablation) | 0.1B ReLU model · Reading comprehension average | accuracy: 28.4 -> 26.2 | Hurt |
Surprising findings
- ReLU models become sparser with more training data, while SiLU models become less sparse with more data.
- Limit activation ratio is weakly dependent on model scale (SiLU drops 2.7 points, ReLU rises 1.7 points from 0.1B to 1.2B).
Limitations
- Sparsity metric depends on the validation dataset and may vary across data distributions and tasks.
- Width-depth ratio experiments omit computation cost; very deep models are much less efficient to train.
- Complex benchmarks (HumanEval, MBPP, GSM8K, MMLU, BBH) are at random level for the small models, so reliable task conclusions are limited.
Experimental setup
Datasets: CommonCrawl Dolma C4 Pile The Stack StarCoder UltraChat SlimOrca OssInstruct EvolInstruct PIQA SIQA
Benchmarks: PIQA SIQA HellaSwag WinoGrande COPA BoolQ LAMBADA TyDi QA AGIEval HumanEval MBPP GSM8K
Metrics: accuracy perplexity pass@1 activation ratio decoding speed (tokens/s)
Baselines: Straightforward ReLU Top-k FAT-epsilon CATS llama.cpp
Builds on: CETT muP PowerInfer FATReLU
Closest papers by meaning
- Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.92)
- Training-Free Activation Sparsity in Large Language Models (ICLR 2025 · similarity 0.92)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.92)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.92)
- ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models (ICLR 2024 · similarity 0.92)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.