AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Sparse is Enough in Scaling Transformers

Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, Jonni Kanerva

NeurIPS 2021 · Architectures and efficiency › Attention and transformers · approach: Sparse efficient transformers for long context · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace dense FF and QKV with sparse FF+QKVScaling Transformer · GLUE validation tasksaccuracy: RTE 70.1± 1.1 -> 68.4; MRPC 83.6± 0.72 -> 81.2; SST-2 92.6± 0.85 -> 91.6; QNLI 88.6± 0.5 -> 90.1; MNLI-m 78.5± 0.41 -> 8Mixed
Use sparse FF+QKV and sparse loss layersTerraformer, 17B · Unbatched CPU decodingspeedup: 1x -> 37.64xImproved
Increase FF sparsity to N=64 (ablation)Scaling Transformer · C4log-perplexityNo clear effect
Increase FF sparsity to N=128 (ablation)Scaling Transformer · C4log-perplexityHurt
Use Mixture-of-Experts-style FF sparsity instead of the proposed sparse FF (ablation)Scaling Transformer · C4log-perplexity: 1.64Hurt
Replace dense QKV layers with sparse QKV layers (ablation)Scaling Transformer · C4log-perplexityNo clear effect
Combine sparse FF and sparse QKV (ablation)Scaling Transformer · Decodingspeedup: 3.05xImproved
Replace dense loss layer with a multiplicative sparse loss layer (ablation)Scaling Transformer · C4log-perplexityHurt
Increase sparse loss modules from baseline to S=4 (ablation)Scaling Transformer · Decodingdecoding time: 0.160 s -> 0.149 sImproved
Add low-rank SRU recurrenceTerraformer · Copy and addition beyond training lengthsfully correct sequences: copy (seq): 0% -> 93.9%; add (seq): 0% -> 32.4%Improved
Store discrete sparse-selection decisions and reuse them during reversible backward passes (ablation)Terraformer · Training with sparse layersmodel performanceImproved
Use a 4x larger batch size with reversible Terraformer training (ablation)Terraformer, 800M · C4log perplexity: 1.66 -> 1.59 across 125k to 175k steps; baseline 1.57Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy