Research map / Architectures and efficiency / Attention and transformers
Sparse Modular Activation for Efficient Sequence Modeling
Liliang Ren, Yang Liu, Shuohang Wang, Yichong Xu, Chenguang Zhu, ChengXiang Zhai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add sparse GAU activation to SSM sequence modeling | SeqBoat versus S5 · LRA | average accuracy: 87.46 -> 87.62 | Improved |
| Add sparse GAU activation to SSM sequence modeling | SeqBoat versus MEGA-chunk · LRA | average accuracy: 85.66 -> 87.62 | Improved |
| Add sparse GAU activation to SSM sequence modeling | SeqBoat versus S4 · LRA | training speedup: 4.8$\times$ -> 10.4$\times$ | Improved |
| Add sparse GAU activation to SSM sequence modeling | SeqBoat versus S4 · Speech Commands SC10 | accuracy: 97.50 -> 97.35 | Hurt |
| Add sparse GAU activation to SSM sequence modeling | SeqBoat versus MEGA-chunk · Speech Commands SC10 | training speedup: 1.00$\times$ -> 1.32$\times$ | Improved |
| Add sparse GAU activation to SSM sequence modeling | SeqBoat versus MEGA-chunk · enwik8 | bits per character: 1.02 -> 1.02 | No clear effect |
| Add working memory with local attention to the compressed sequence | SeqBoat versus SeqBoat-full · LRA Pathfinder | accuracy: 93.00 -> 96.35 | Improved |
| Always activate GAU instead of using SMA (ablation) | SeqBoat-full · LRA Image | validation accuracy: 90.16 -> 90.70 | Improved |
| Always activate GAU instead of using SMA (ablation) | SeqBoat-full · LRA ListOps | training speedup: 1.79$\times$ -> 1.11$\times$ | Hurt |
| Replace Tempered Softmax with Gumbel Softmax (ablation) | SeqBoat-full · LRA Image | validation accuracy: 90.16 -> 89.94 | Hurt |
| Replace Tempered Softmax with Gumbel Softmax (ablation) | SeqBoat-full · LRA ListOps | validation accuracy: 60.95 -> 59.75 | Hurt |
| Replace MD-EMA kernel with S4D (ablation) | SeqBoat-full · LRA Image | validation accuracy: 90.16 -> 82.84 | Hurt |
Closest papers by meaning
- FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.87)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.87)
- Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs (CVPR 2024 · similarity 0.87)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.86)
- Loki: Low-rank Keys for Efficient Sparse Attention (NeurIPS 2024 · similarity 0.86)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
