Research map / Architectures and efficiency / Attention and transformers
Bridging the Divide: Reconsidering Softmax and Linear Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why linear attention lags Softmax attention, identifying non-injectivity and weak local modeling as key causes, and proposes InLine attention (subtraction-based normalization plus a local attention residual) that outperforms Softmax counterparts on ImageNet, COCO, and ADE20K.
Problem
Linear attention substantially underperforms Softmax attention in vision Transformers, and the fundamental reasons for this gap are not well understood.
Approach
Shows Softmax attention is injective while linear attention (with division normalization) is not, and that Softmax relies on strong local modeling; proposes InLine attention, which replaces division normalization with subtraction so the attention function is injective, plus an MLP-predicted local attention residual over 3x3 neighborhoods to restore local bias.
Key results
- InLine-Swin-T improves over Swin-T on ImageNet-1K from 81.3 to 82.4 top-1 at 4.5G FLOPs.
- InLine-PVT-S reaches 82.0 top-1 vs. PVT-L at 81.7 with about 30% of the parameters and 40% of the FLOPs.
- Swapping ReLU-kernel linear attention for InLine on Swin-T raises accuracy from 70.2 to 80.0 with the ReLU(Aq+b) kernel.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Softmax attention with InLine attention | CSwin-T · ImageNet-1K | Top-1 accuracy: 82.7 -> 83.2 | Helped |
| Replace Softmax attention with InLine attention | CSwin-B · ImageNet-1K | Top-1 accuracy: 84.2 -> 84.5 | Helped |
| Replace Softmax attention with InLine attention | CSwin-B · ImageNet-1K, 384² | Top-1 accuracy: 85.4 -> 85.7 | Helped |
| Replace the PVT attention with InLine attention | PVT-L · COCO, Mask R-CNN, 1x | box AP: 42.9 -> 45.4 | Helped |
| Replace the Swin attention with InLine attention | Swin-B · COCO, Cascade Mask R-CNN, 3x | box AP: 51.9 -> 52.6 | Helped |
| Replace the PVT attention with InLine attention | PVT-L · ADE20K, SemanticFPN | mIoU: 43.49 -> 44.71 | Helped |
| Replace the Swin attention with InLine attention | Swin-B · ADE20K, UperNet | mIoU: 48.13 -> 49.10 | Helped |
| Add injective InLine attention instead of normalized linear attention (ablation) | Swin-T with ReLU kernel · ImageNet-1K | accuracy: 77.3 -> 79.8 | Helped |
| Add injective InLine attention instead of normalized linear attention (ablation) | Swin-T with ReLU(A·+b) kernel · ImageNet-1K | accuracy: 70.2 -> 80.0 | Helped |
| Add a local attention residual (ablation) | InLine-Swin-T · ImageNet-1K | accuracy: 80.3 -> 81.6; 80.4 -> 82.1; 80.2 -> 82.3; 80.2 -> 82.4 | Helped |
| Increase the window size from 7² to 56² without local residual (ablation) | InLine-Swin-T · ImageNet-1K | accuracy: 80.3; 80.4; 80.2; 80.2 | No clear effect |
| Use a ReLU kernel instead of Identity (ablation) | InLine-Swin-T · ImageNet-1K | accuracy: 82.4 -> 82.5 | No clear effect |
Surprising findings
- Injecting non-injective confusion into Softmax attention (normalized ReLU queries) drops Swin accuracy from 72.2 to 70.6 or 69.9 on DeiT-T, showing injectivity matters even for Softmax.
- Larger window sizes do not improve pure InLine attention without the local residual, indicating insufficient local modeling.
Limitations
- The paper acknowledges other differences between Softmax and linear attention may exist and the analysis is not exhaustive.
- Gains from InLine are reported on vision backbones only; no evaluation on language or other domains.
- Injectivity proofs rely on mild rank assumptions on key matrices that are argued plausible but not verified in general.
Experimental setup
Datasets: ImageNet-1K COCO ADE20K
Benchmarks: ImageNet-1K COCO ADE20K
Metrics: top-1 accuracy box AP mask AP mIoU FLOPs throughput
Baselines: DeiT PVT Swin Transformer CSwin FLatten Hydra Attention Efficient Attention Linear Angular Attention ConvNeXt MViTv2 NAT DiNAT
Builds on: Swin Transformer Linear attention (Katharopoulos et al.) CSwin Transformer PVT DeiT
Closest papers by meaning
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.92)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.90)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.89)
- MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map (NeurIPS 2024 · similarity 0.89)
- ZeroS: Zero‑Sum Linear Attention for Efficient Transformers (NeurIPS 2025 · similarity 0.89)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.