Research map / Architectures and efficiency / Attention and transformers
Simplified State Space Layers for Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace many independent SISO S4 SSMs with one MIMO S5 SSM using diagonalized dynamics and parallel scans | S5 layer · Long Range Arena | Avg. accuracy: 87.46 vs 86.09 for S4-LegS | Helped |
| Use S5 instead of S4-LegS | Path-X task · LRA Path-X | accuracy: 98.58 vs 96.35 | Helped |
| Use S5 instead of S4 on speech classification at 0-shot 8kHz | Speech Commands · 8kHz zero-shot testing | accuracy: 94.53 vs 91.32 for S4-LegS | Helped |
| Use S5 instead of S4-LegS on 16kHz speech | Speech Commands · 16kHz | accuracy: 96.52 vs 96.08 for S4-LegS; below Liquid-S4 96.78 | Mixed |
| Use S5 instead of CRU on irregularly sampled pendulum regression | Pendulum regression · Pendulum regression | regression MSE x10^-3: 3.41 vs 3.94 for CRU (our run) | Helped |
| Initialize S5 state matrix with diagonalized HiPPO-N instead of exact HiPPO-LegS | S5 layer · Long Range Arena | qualitative | No clear effect |
| Set S5 latent size P equal to H (P=H) instead of P=N in runtime comparison (ablation) | S5 layer · Training step speed on Path-X | relative train step speed: 1.9x vs 2.9x for P=N | No clear effect |
Closest papers by meaning
- State Space Models are Provably Comparable to Transformers in Dynamic Token Selection (ICLR 2025 · similarity 0.87)
- Longhorn: State Space Models are Amortized Online Learners (ICLR 2025 · similarity 0.86)
- From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics (ICLR 2025 · similarity 0.86)
- Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · similarity 0.86)
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.86)
- Layer-Adaptive State Pruning for Deep State Space Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.