Research map / Architectures and efficiency / Attention and transformers
cosFormer: Rethinking Softmax In Attention
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, Yiran Zhong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace softmax attention with cosFormer linear attention | autoregressive language modeling · WikiText-103 test | perplexity: 26.2 -> 23.1 | Improved |
| Replace softmax attention with cosFormer linear attention | bidirectional text classification model · QQP | score: 88.41 -> 89.26 | Improved |
| Replace softmax attention with cosFormer linear attention | bidirectional text classification model · SST-2 | score: 92.31 -> 91.05 | Hurt |
| Replace softmax attention with cosFormer linear attention | bidirectional text classification model · MNLI | score: 79.15 -> 76.70; Longformer: 77.22 | Mixed |
| Replace softmax attention with cosFormer linear attention | bidirectional text classification model · IMDB | score: 92.86 -> 92.95 | Improved |
| Replace softmax attention with cosFormer linear attention | bidirectional text classification model · AMAZON | score: 75.79 -> 76.30 | Improved |
| Replace softmax attention with cosFormer linear attention | Long-Range Arena model · Long-Range Arena average | average score: 54.39 -> 55.23 | Improved |
| Replace softmax attention with cosFormer linear attention | Long-Range Arena model · Long-Range Arena Pathfinder | score: 77.05 -> 70.33 | Hurt |
| Add cosine-based re-weighting (ablation) | cosFormer · Bidirectional fine-tuning average | average score: 85.12 -> 85.25 | Improved |
| Add cosine-based re-weighting (ablation) | cosFormer · Long-Range Arena average | average score: 54.20 -> 55.23 | Improved |
| Use ReLU features instead of identity mapping (ablation) | linear attention pre-trained on WikiText-103 · QQP | accuracy: 84.23 -> 88.86 | Improved |
| Use ReLU features instead of LeakyReLU features (ablation) | linear attention pre-trained on WikiText-103 · MNLI | accuracy: 74.26 -> 77.86 | Improved |
Later papers that compared with this method
6 later accepted papers report CosFormer as a baseline in their results tables; 5 of them report a better number on most of the results they share with it.
- ZETA: Leveraging $Z$-order Curves for Efficient Top-$k$ Attention (ICLR 2025 · better on 0 of 1 shared result)
- MixA: A Mixed Attention approach with Stable Lightweight Linear Attention to enhance Efficiency of Vision Transformers at the Edge (ICCV 2025 · better on 1 of 1 shared result)
- The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · better on 1 of 1 shared result)
- Linear Log-Normal Attention with Unbiased Concentration (ICLR 2024 · better on 1 of 1 shared result)
- LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions (ICML 2024 · better on 1 of 1 shared result)
- Flowformer: Linearizing Transformers with Conservation Flows (ICML 2022 · better on 1 of 1 shared result)
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- BigBird — Big Bird: Transformers for Longer Sequences (NeurIPS 2020) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.88)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.88)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
- SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024 · similarity 0.87)
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.87)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
