Research map / Architectures and efficiency / Attention and transformers
Demystifying the Token Dynamics of Deep Selective State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes token dynamics of a pre-trained Mamba (S6 layer) via its continuous-depth limit in the 1D case, proving tokens either all converge to zero or all diverge, and uses the findings to propose excluding the convergent regime and reordering tokens by importance scores.
Problem
Theoretical understanding of deep selective state space models like Mamba is lacking; how tokens evolve across layers and how this affects performance is unclear.
Approach
Treat Mamba's layer index as continuous time to derive a dynamical system for S6 token evolution, classify asymptotic behavior by the sign of S_C^T S_B and S_Delta x_l0, and turn the findings into two training refinements (excluding the convergent regime, reordering tokens by learned importance scores).
Key results
- On WikiText103, perplexity is 16.66 for the all-positive-eigenvalue scenario versus 17.28 for the all-negative scenario and 16.82 for mixed.
- Token reordering on MambaVision-T improves ImageNet-1K top-1 accuracy from 81.90% to 82.02% (top-5 95.86% to 95.87%).
- Theory: in 1D, tokens either all converge to zero (S_C^T S_B<0, O(1/sqrt t)) or all diverge (slow log-rate or finite-time blow-up).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reorder tokens by ascending importance scores | MambaVision-T · ImageNet-1K | Top-1 accuracy: 81.90 -> 82.02 | Helped |
| Reorder tokens by ascending importance scores | MambaVision-T · ImageNet-1K | Top-5 accuracy: 95.86 -> 95.87 | Helped |
| Use only positive eigenvalues instead of only negative eigenvalues in the input-output matrix | Mamba · WikiText103 test | Perplexity: 17.28 -> 16.66 | Helped |
| Use only positive eigenvalues instead of both positive and negative eigenvalues in the input-output matrix | Mamba · WikiText103 test | Perplexity: 16.82 -> 16.66 | Helped |
| Reorder tokens by importance scores | MambaVision-T · ImageNet-1K training | Convergence speed | Helped |
Surprising findings
- Convergent regime (negative input-output product) collapses tokens and attention to zero and hurts performance, the opposite of the usual intuition that stability helps.
Limitations
- Theory restricted to one feature channel (D=1); higher-dimensional case only conjectured.
- Ignores layer norm, convolution, and linear layers, and assumes time-independent parameters.
- Improvement from token reordering is small (0.12 points top-1) and optimality of the learnable-line ordering is uncertain.
Experimental setup
Datasets: WikiText103 ImageNet-1K
Benchmarks: WikiText103 ImageNet-1K
Metrics: perplexity top-1 accuracy top-5 accuracy
Baselines: Mamba MambaVision-T
Builds on: Mamba neural ODEs SoftSort MambaVision
Closest papers by meaning
- Theoretical Foundations of Deep Selective State-Space Models (NeurIPS 2024 · similarity 0.90)
- State Space Models are Provably Comparable to Transformers in Dynamic Token Selection (ICLR 2025 · similarity 0.89)
- Block-Biased Mamba for Long-Range Sequence Processing (NeurIPS 2025 · similarity 0.89)
- Swimba: Switch Mamba Model Scales State Space Models (NeurIPS 2026 · similarity 0.88)
- TIDES: Implicit Time-Awareness in Selective State Space Models (NeurIPS 2026 · similarity 0.88)
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.