Research map / Architectures and efficiency / Attention and transformers
Efficiently Modeling Long Sequences with Structured State Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use S4 instead of efficient Transformer baselines | sequence modeling on LRA · LRA, six tasks | accuracy: 86.09 vs. 59.37 | Helped |
| Use S4 on Path-X instead of prior models | long-sequence classification · LRA Path-X | accuracy: 88% accuracy compared to 50% random guessing | Helped |
| Use S4 for raw audio instead of specialized Speech CNNs | SC10 raw audio classification · Speech Commands SC10 | test error: 1.7% | Helped |
| Use S4 for long-sequence forecasting instead of Informer | univariate forecasting · ETTh1, ETTh2, ETTm1, Weather, ECL | MSE: 0.116 vs. 0.269; 0.187 vs. 0.277; 0.292 vs. 0.512; 0.245 vs. 0.359; 0.432 vs. 0.582 | Helped |
| Initialize the SSM with HiPPO rather than random initialization (ablation) | trainable SSMs · Sequential CIFAR-10 | validation accuracy | Helped |
| Train the SSM rather than freeze its state matrix (ablation) | SSMs with different initializations · Sequential CIFAR-10 | validation accuracy | Helped |
| Use HiPPO initialization instead of random initialization in an NPLR SSM (ablation) | NPLR SSM · Sequential CIFAR-10 | validation accuracy | Helped |
| Use HiPPO initialization and 0.1 Dropout (ablation) | S4 · Sequential CIFAR-10, 100K parameters | test accuracy: 84.27% | Helped |
| Use S4 instead of ResNet18 without data augmentation | sequential image classification · Sequential CIFAR-10 | accuracy: 89.46% vs. 91.12% | Helped |
| Use S4 instead of ResNet18 with data augmentation | sequential image classification · Sequential CIFAR-10 | accuracy: 93.16% vs. 95.62% | Hurt |
| Swap BatchNorm for LayerNorm (ablation) | S4 and ResNet18 · Sequential CIFAR-10 | accuracy: 90.46% vs. 79.52% | Mixed |
Closest papers by meaning
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.88)
- Longhorn: State Space Models are Amortized Online Learners (ICLR 2025 · similarity 0.88)
- Block-Biased Mamba for Long-Range Sequence Processing (NeurIPS 2025 · similarity 0.88)
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.88)
- Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.87)
- Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.