Research map / Architectures and efficiency / Attention and transformers
SOFT: Softmax-free Transformer with Linear Complexity
Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, Li Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace softmax dot-product attention with Gaussian-kernel SOFT attention | Tiny Transformer · ImageNet-1K validation | Top-1 accuracy (%): 78.2 -> 79.3 | Improved |
| Replace the PVT attention block with SOFT attention | PVT-style models · ImageNet-1K validation, model variants | Top-1 accuracy (%): 75.1 -> 79.3; 79.8 -> 82.9; 81.2 -> 82.9; 81.7 -> 83.1 | Improved |
| Use SOFT on LRA tasks instead of Nyströmformer | Efficient Transformer · LRA Retrieval (4K) | score: 79.56 -> 81.77 | Improved |
| Use SOFT instead of Nyströmformer | Efficient Transformer · LRA Image (1K) | score: 41.58 -> 46.91 | Improved |
| Use SOFT instead of Nyströmformer | Efficient Transformer · LRA Text (4K) | score: 65.52 -> 63.49 | Hurt |
| Use bottleneck length 64 instead of 49 (ablation) | SOFT-Tiny · ImageNet-1K validation | Top-1 accuracy (%): 79.3 -> 79.3 | No clear effect |
| Use bottleneck length 81 instead of 49 (ablation) | SOFT-Tiny · ImageNet-1K validation | Top-1 accuracy (%): 79.3 -> 78.9 | Hurt |
| Use bottleneck length 36 instead of 49 (ablation) | SOFT-Tiny · ImageNet-1K validation | Top-1 accuracy (%): 79.3 -> 79.0 | Hurt |
| Use biased sampling instead of convolution sampling (ablation) | SOFT-Tiny · ImageNet-1K validation | Top-1 accuracy (%): 79.3 -> 79.0 | Hurt |
| Use average pooling instead of convolution sampling (ablation) | SOFT-Tiny · ImageNet-1K validation | Top-1 accuracy (%): 79.3 -> 79.3 | No clear effect |
| Add a pyramidal architecture (ablation) | SOFT-Small · ImageNet-1K validation | Top-1 accuracy (%): 80.1 -> 82.2 | Improved |
| Add overlapping convolution (ablation) | SOFT-Tiny · ImageNet-1K validation | Top-1 accuracy (%): 77.4 -> 79.3 | Improved |
Later papers that compared with this method
2 later accepted papers report SOFT++-Medium as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.
- LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel (ICLR 2026 · better on 12 of 12 shared results)
- Vicinity Vision Transformer (TPAMI 2023 · better on 1 of 1 shared result)
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- Nyströmformer — Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention (AAAI 2021) · better on 4 of 5 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.89)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.88)
- Transformer-VQ: Linear-Time Transformers via Vector Quantization (ICLR 2024 · similarity 0.88)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.88)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.88)
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
