Research map / Architectures and efficiency / Attention and transformers
Theoretical Foundations of Deep Selective State-Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a rough-path/signature-based expressivity characterization of gated linear-in-state SSMs (linear CDEs), showing diagonal gated recurrences like Mamba have strictly smaller closure than dense ones, with chaining recovering expressivity.
Problem
Theoretical understanding of what functions selective (input-dependent, gated) state-space models such as Mamba can express, compared with S4 and dense linear recurrences.
Approach
Casts a broad family of SSMs (S4, Mamba, GLA-type gated recurrences) as linear controlled differential equations driven by gate paths, then uses rough path theory and the signature transform to fully characterize their uniform closure; shows diagonal (Mamba-style) recurrences are strictly less expressive than dense ones and that stacking/chaining diagonal CDEs recovers full expressivity.
Key results
- On the A5 benchmark, RNN and Linear CDE exceed 90% validation accuracy for all lengths with one block, while S4, Mamba, and Transformer fail to reach 90% at length 20 within 4 blocks.
- On the anti-symmetric signature task, stacking diagonal Mamba layers improves expressivity, while dense Linear CDE with random fixed matrices and a trained linear readout attains strong performance with no stacking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a dense transition recurrence instead of a diagonal recurrence | Linear CDE · A5 state tracking | validation accuracy | Helped |
| Stack diagonal SSM recurrences (ablation) | Mamba · anti-symmetric signature prediction | validation accuracy | Helped |
| Add a non-linearity between stacked Mamba layers (ablation) | Mamba · anti-symmetric signature prediction | validation accuracy | No clear effect |
| Add a non-linearity between stacked S4 layers (ablation) | S4 · anti-symmetric signature prediction | validation accuracy | Helped |
| Use a dense Linear CDE with random fixed recurrence matrices and a trainable linear readout, without stacking | Linear CDE · anti-symmetric signature prediction | validation accuracy | Helped |
| Use one block rather than a deeper stack | Linear CDE · A5, sequence lengths 3–20 | validation accuracy | Helped |
| Use one block rather than a deeper stack | RNN · A5, sequence lengths 3–20 | validation accuracy | Helped |
| Use diagonal recurrence rather than dense transition matrices | SSM · A5, sequence length 20 | validation accuracy | Hurt |
Surprising findings
- Adding a GeLU non-linearity between stacked Mamba layers does not improve performance on the signature task.
- Linear-in-input S4 requires a non-linearity between layers to perform well on the signature task.
Limitations
- Analysis is continuous-time with real-valued inputs and ignores finite precision effects.
- Dense transition matrices are computationally impractical for large-scale models.
- Experiments are small synthetic tasks with fixed architecture, not large-scale language modeling.
Experimental setup
Datasets: anti-symmetric signature prediction (synthetic, 2D and 3D) A5 state-tracking benchmark
Benchmarks: anti-symmetric signature prediction A5 state tracking
Metrics: validation accuracy minimum number of blocks to reach 90% validation accuracy
Baselines: S4 Mamba RNN Transformer stacked S4 stacked Mamba
Builds on: S4 Mamba Universal approximation for linear RNNs Signature transform
Closest papers by meaning
- Demystifying the Token Dynamics of Deep Selective State Space Models (ICLR 2025 · similarity 0.90)
- On the Expressiveness and Length Generalization of Selective State Space Models on Regular Languages (AAAI 2025 · similarity 0.89)
- Structural Inference of Dynamical Systems with Conjoined State Space Models (NeurIPS 2024 · similarity 0.88)
- Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention (NeurIPS 2025 · similarity 0.88)
- State Space Models are Provably Comparable to Transformers in Dynamic Token Selection (ICLR 2025 · similarity 0.88)
- TIDES: Implicit Time-Awareness in Selective State Space Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.