Research map / Architectures and efficiency / Attention and transformers
Interpreting and Steering State-Space Models via Activation Subspace Bottlenecks
Vamshi Sunku Mohan, Kaustubh Gupta, Aneesha Das, Chandan Singh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| steer delta-sensitive subspaces at Layer 20 (scale activations by scalar) | Mamba-130M · RULER NIAH | accuracy (%): 84.00 -> 92.00 | Improved |
| steer delta-sensitive subspaces at Layer 20 | Mamba-130M · RULER QA | accuracy (%): 66.70 -> 76.00 | Improved |
| steer delta-sensitive subspaces at Layer 20 | Mamba-130M · Long Range Arena AVG | average (%): 52.00 -> 60.88 | Improved |
| steer delta-sensitive subspaces at Layer 20 | 7 SSMs across 6 benchmarks · TriviaQA, SQuAD, MuSiQue, IFEval, RULER, DROP | average improvement (%): +8.27% | Improved |
| steer Layer 20 instead of other layers (ablation) | Mamba-130M steering · IFEval | accuracy (%): 59.0 -> 72.5 (Layer 21 vs Layer 20) | Improved |
| select delta-sensitive subspaces instead of random subspaces (ablation) | Mamba-130M steering · IFEval | accuracy (%): 67.0 -> 72.5 | Improved |
| select delta-sensitive instead of high-variance subspaces (ablation) | Mamba-130M steering · IFEval | accuracy (%): 33.5 -> 72.5 | Improved |
| steer delta-sensitive instead of uniform steering across subspaces (ablation) | Mamba-130M steering · IFEval | accuracy (%): 61.5 -> 72.5 | Improved |
| retrain with Stable-Mamba architecture | Mamba-130M · Long Range Arena PathFinder | accuracy (%): 60.00 -> 95.00 | Improved |
| retrain with Stable-Mamba architecture | Mamba-130M · RULER NIAH | accuracy (%): 84.00 -> 90.00 | Improved |
| retrain with Stable-Mamba architecture | Mamba-130M · RULER QA | accuracy (%): 66.70 -> 82.00 | Improved |
| retrain with Stable-Mamba architecture | Mamba-130M · SAE sparsity | sparsity (%): 98.12 -> 58.37 | Improved |
Closest papers by meaning
- The Expressive Capacity of State Space Models: A Formal Language Perspective (NeurIPS 2024 · similarity 0.90)
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (ICML 2024 · similarity 0.90)
- Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.89)
- LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement (ICLR 2025 · similarity 0.89)
- Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing (ICLR 2025 · similarity 0.88)
- The Illusion of State in State-Space Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
