Research map / Architectures and efficiency / Attention and transformers
Sparse is Enough in Scaling Transformers
Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, Jonni Kanerva
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dense FF and QKV with sparse FF+QKV | Scaling Transformer · GLUE validation tasks | accuracy: RTE 70.1± 1.1 -> 68.4; MRPC 83.6± 0.72 -> 81.2; SST-2 92.6± 0.85 -> 91.6; QNLI 88.6± 0.5 -> 90.1; MNLI-m 78.5± 0.41 -> 8 | Mixed |
| Use sparse FF+QKV and sparse loss layers | Terraformer, 17B · Unbatched CPU decoding | speedup: 1x -> 37.64x | Improved |
| Increase FF sparsity to N=64 (ablation) | Scaling Transformer · C4 | log-perplexity | No clear effect |
| Increase FF sparsity to N=128 (ablation) | Scaling Transformer · C4 | log-perplexity | Hurt |
| Use Mixture-of-Experts-style FF sparsity instead of the proposed sparse FF (ablation) | Scaling Transformer · C4 | log-perplexity: 1.64 | Hurt |
| Replace dense QKV layers with sparse QKV layers (ablation) | Scaling Transformer · C4 | log-perplexity | No clear effect |
| Combine sparse FF and sparse QKV (ablation) | Scaling Transformer · Decoding | speedup: 3.05x | Improved |
| Replace dense loss layer with a multiplicative sparse loss layer (ablation) | Scaling Transformer · C4 | log-perplexity | Hurt |
| Increase sparse loss modules from baseline to S=4 (ablation) | Scaling Transformer · Decoding | decoding time: 0.160 s -> 0.149 s | Improved |
| Add low-rank SRU recurrence | Terraformer · Copy and addition beyond training lengths | fully correct sequences: copy (seq): 0% -> 93.9%; add (seq): 0% -> 32.4% | Improved |
| Store discrete sparse-selection decisions and reuse them during reversible backward passes (ablation) | Terraformer · Training with sparse layers | model performance | Improved |
| Use a 4x larger batch size with reversible Terraformer training (ablation) | Terraformer, 800M · C4 | log perplexity: 1.66 -> 1.59 across 125k to 175k steps; baseline 1.57 | Mixed |
Closest papers by meaning
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.89)
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.88)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.88)
- Ultra-Sparse Memory Network (ICLR 2025 · similarity 0.87)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.87)
- Sparse Layers are Critical to Scaling Looped Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
