Research map / Architectures and efficiency / Attention and transformers
Scaling Linear Attention with Sparse State Expansion
Yuqi Pan, Yongqi An, Zheng Li, Yuhong Chou, Rui-Jie Zhu, Xiaohui Wang, Mingxuan Wang, Jinqiao Wang, Guoqi Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use softmax instead of SiLU as key feature map (ablation) | SSE-n4k1 (600M) · FDA/SWDE/SQuAD recall avg | accuracy: 24.23 -> 31.16 | Improved |
| add write-read gate on both KV and Q (ablation) | SSE-n4k1 (2B) · Recall avg | accuracy: 51.87 -> 56.63 | Improved |
| add write-only gate vs no gate (ablation) | SSE-n4k1 (2B) · Recall avg | accuracy: 51.87 -> 50.92 | No clear effect |
| add read-only gate vs no gate (ablation) | SSE-n4k1 (2B) · Recall avg | accuracy: 51.87 -> 53.96 | Improved |
| share attention parameters across partitions (ablation) | SSE-n4k1 (600M) · Recall avg | accuracy: 25.95 -> 31.16 | Improved |
| increase number of partitions n at constant sparsity ratio (ablation) | SSE (2B, 20B tokens) · recall | recall accuracy | Improved |
| increase sparsity ratio k/n beyond about 50% (ablation) | SSE (2B, 20B tokens) · recall | recall accuracy | No clear effect |
| add sparse state expansion (n4k1) to GLA-style linear attention | GLA (2B) · Real-world recall avg | accuracy: 49.29 -> 61.46 | Improved |
| add one softmax layer per five linear layers to SSE (hybrid) | SSE-n4k1 (2B) · Real-world recall avg | accuracy: 61.46 -> 70.87 | Improved |
| use SSE-H hybrid instead of GLA-H | 2B hybrid · RULER S-NIAH-3 at 8K | accuracy: 70.8 -> 97.4 | Improved |
| use SSE-H instead of Transformer after 2T tokens plus long-context extension | 2B · MMLU | accuracy: 52.6 -> 54.5 | Improved |
| use SSE-H instead of Transformer for reasoning pipeline | 2B · AIME25 | avg@32 accuracy: 52.8 -> 50.2 | No clear effect |
Closest papers by meaning
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.89)
- Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity (NeurIPS 2026 · similarity 0.89)
- Overcoming Long Context Limitations of State Space Models via Context Dependent Sparse Attention (NeurIPS 2025 · similarity 0.89)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.88)
- SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024 · similarity 0.88)
- Block Sparse Flash Attention (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
