Research map / Architectures and efficiency / Attention and transformers
On Structured State-Space Duality
Jerry Yao-Chieh Hu, Xiwen Zhang, Ali ElSheikh, Weimin Wu, Han Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| rewrite scalar SSM as 1-SS masked attention | scalar SSM · synthetic sequences, 1000 seeds | max absolute error: error below 10-14 | No clear effect |
| rewrite diagonal SSM as sum of N 1-SS attention heads | diagonal SSM (N=2) · synthetic sequences, 1000 seeds | max absolute error: error on the order of 10-14 | No clear effect |
| use duplicated decays {0.7, 0.7} instead of distinct decays (ablation) | diagonal SSM kernel · synthetic, N=2 | semiseparable rank: rank 2 -> 1 | Hurt |
Closest papers by meaning
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (ICML 2024 · similarity 0.89)
- Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks (NeurIPS 2024 · similarity 0.88)
- Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences (ICML 2024 · similarity 0.87)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.87)
- Theoretical Foundations of Deep Selective State-Space Models (NeurIPS 2024 · similarity 0.87)
- Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
