Research map / Large language models / Reasoning and chain of thought
ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows ReLU matches SiLU/GELU in LLM performance when trained from scratch or after finetuning ("relufication"), cutting inference FLOPS up to ~3x via activation sparsity, and proposes aggregated sparsity reuse and shifted ReLU for further gains.
Problem
LLM inference is computationally and memory-bound, and most modern LLMs use smooth non-ReLU activations (SiLU, GELU) that produce dense activations, forfeiting activation sparsity-based savings.
Approach
Replace SiLU/GELU with ReLU (relufication) in pretrained LLMs and briefly finetune to recover performance, then insert extra ReLUs after normalization layers to sparsify QKV and up-projection inputs; additionally exploit neuron reuse across tokens (aggregated sparsity) and shifted ReLU ReLU(x-b) to increase sparsity.
Key results
- Relufied Falcon 7B (stage 2) reaches 64.8% zero-shot average with 2.2 GFLOPS versus 66.8% at 6.6 GFLOPS for the original Falcon 7B.
- Relufied OPT 6.7B (stage 2) reaches 58.6% average at 2.8 GFLOPS versus 59.8% at 4.5 GFLOPS for the original OPT 6.7B.
- Sparse speculative decoding with aggregated sparsity gives a 1.27x speedup over standard speculative decoding at gamma=16 for OPT 6.7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GELU with ReLU in Falcon 7B FFN (stage 1) (ablation) | Falcon 7B · Zero-shot tasks (average) | FLOPS (G): 6.6 -> 4.1 | Helped |
| replace GELU with ReLU in Falcon 7B FFN (stage 1) (ablation) | Falcon 7B · Zero-shot tasks (average) | accuracy (%): 66.8 -> 65.2 | Hurt |
| add ReLU after normalization layers (stage 2) (ablation) | Falcon 7B (s1) · Zero-shot tasks (average) | FLOPS (G): 4.1 -> 2.2 | Helped |
| add ReLU after normalization layers (stage 2) (ablation) | Falcon 7B (s1) · Zero-shot tasks (average) | accuracy (%): 65.2 -> 64.8 | Hurt |
| replace SiLU with ReLU in Llama 7B FFN (stage 1) (ablation) | Llama 7B · Zero-shot tasks (average) | FLOPS (G): 6.6 -> 4.8 | Helped |
| add ReLU after normalization layers (stage 2) (ablation) | Llama 7B (s1) · Zero-shot tasks (average) | FLOPS (G): 4.8 -> 2.9 | Helped |
| replace SiLU with ReLU in Llama 7B FFN (stage 1) (ablation) | Llama 7B · Zero-shot tasks (average) | accuracy (%): 68.4 -> 67.1 | Hurt |
| add ReLU after normalization layers (stage 2) (ablation) | Llama 7B (s1) · Zero-shot tasks (average) | accuracy (%): 67.1 -> 66.4 | Hurt |
| use ReLU instead of SiLU or GELU | Falcon 7B · MMLU five-shot | average accuracy: 26.4; 27.7; 27.9 | No clear effect |
| use ReLU instead of SiLU or GELU | Llama 7B · MMLU five-shot | average accuracy: 35.1; 35.9; 34.7 | No clear effect |
| use aggregated sparsity in speculative decoding rather than random sparsity | OPT 6.7B speculative decoding, γ=16 · Speculative decoding | speedup over standard speculative decoding: 1.27x; 1.20x | Helped |
| reuse previously activated neurons rather than randomly selected activations | ReLU LLM token generation · Perplexity | perplexity | Helped |
Surprising findings
- Inserting ReLU after normalization layers makes the QKV inputs sparse (roughly 50-56%) even though they were not sparse before, enabling further FLOPS reduction.
- The relufied Llama model has only about 65% sparsity versus about 95% for relufied Falcon, which motivated the shifted ReLU.
Limitations
- FLOPS is used as a proxy for efficiency; real latency was not directly measured for the main results.
- Finetuning budget (30-50B tokens) is small relative to original pretraining, so the small performance gap may not be fully closed.
- Results on aggregated sparsity and shifted ReLU are preliminary, with the shifted ReLU applied only to Llama.
Experimental setup
Datasets: RefinedWeb WikiText
Benchmarks: ARC-Easy ARC-Challenge HellaSwag BoolQ PIQA LAMBADA TriviaQA WinoGrande SciQ MMLU
Metrics: zero-shot accuracy 5-shot accuracy FLOPS activation sparsity perplexity
Baselines: GELU SiLU original pretrained models (Llama 7B, Falcon 7B, OPT) dense smaller OPT models
Builds on: speculative decoding Deja Vu SwiGLU MoEfication
Closest papers by meaning
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.92)
- Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.91)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.91)
- Training-Free Activation Sparsity in Large Language Models (ICLR 2025 · similarity 0.90)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.89)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.