AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

cosFormer: Rethinking Softmax In Attention

Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, Yiran Zhong

ICLR 2022 · Architectures and efficiency › Attention and transformers · approach: Linear attention approximations · 65 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace softmax attention with cosFormer linear attentionautoregressive language modeling · WikiText-103 testperplexity: 26.2 -> 23.1Improved
Replace softmax attention with cosFormer linear attentionbidirectional text classification model · QQPscore: 88.41 -> 89.26Improved
Replace softmax attention with cosFormer linear attentionbidirectional text classification model · SST-2score: 92.31 -> 91.05Hurt
Replace softmax attention with cosFormer linear attentionbidirectional text classification model · MNLIscore: 79.15 -> 76.70; Longformer: 77.22Mixed
Replace softmax attention with cosFormer linear attentionbidirectional text classification model · IMDBscore: 92.86 -> 92.95Improved
Replace softmax attention with cosFormer linear attentionbidirectional text classification model · AMAZONscore: 75.79 -> 76.30Improved
Replace softmax attention with cosFormer linear attentionLong-Range Arena model · Long-Range Arena averageaverage score: 54.39 -> 55.23Improved
Replace softmax attention with cosFormer linear attentionLong-Range Arena model · Long-Range Arena Pathfinderscore: 77.05 -> 70.33Hurt
Add cosine-based re-weighting (ablation)cosFormer · Bidirectional fine-tuning averageaverage score: 85.12 -> 85.25Improved
Add cosine-based re-weighting (ablation)cosFormer · Long-Range Arena averageaverage score: 54.20 -> 55.23Improved
Use ReLU features instead of identity mapping (ablation)linear attention pre-trained on WikiText-103 · QQPaccuracy: 84.23 -> 88.86Improved
Use ReLU features instead of LeakyReLU features (ablation)linear attention pre-trained on WikiText-103 · MNLIaccuracy: 74.26 -> 77.86Improved

Later papers that compared with this method

6 later accepted papers report CosFormer as a baseline in their results tables; 5 of them report a better number on most of the results they share with it.

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy