Research map / Architectures and efficiency / Pruning and efficient tuning
SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot
Kaiwen Tuo, Huan Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SparseSSM time-weighted OBS scoring instead of SparseGPT for A_log pruning at 50% sparsity | Mamba-370M · WikiText-2 | perplexity: 2696 -> 15.04 | Improved |
| Use SparseSSM instead of Wanda for A_log pruning at 50% sparsity | Mamba-130M · WikiText-2 | perplexity: 77.84 -> 21.26 | Improved |
| Use SparseSSM instead of SparseGPT | Mamba-370M · zero-shot average of 5 tasks | average accuracy (%): 44.56 -> 47.47 | Improved |
| Prune A_log of Mamba-2.8B to 50% sparsity with SparseSSM | Mamba-2.8B · WikiText-2 | perplexity: 9.45 -> 10.63 | Mixed |
| Replace power-law time-weighted aggregation with simple L2 aggregation over time steps (ablation) | SparseSSM · Mamba-370M zero-shot accuracy | accuracy | Improved |
| Add sensitivity-aware sparsity allocation (alpha=0.04) to FFN pruning (ablation) | SparseSSM on FFN modules · Mamba-370M zero-shot accuracy | accuracy | Improved |
| Use fewer than 16 calibration samples (ablation) | SparseSSM · Mamba-370M SSM pruning | accuracy | Hurt |
| Use 64 calibration samples (ablation) | SparseSSM · Mamba-370M SSM pruning | accuracy and pruning time | Improved |
| Use N:M (2:4) sparsity with SparseSSM instead of magnitude pruning | Mamba-370M SSM · WikiText-2 | perplexity: 77.20 -> 16.90 | Improved |
| Structured column pruning of A_log at 50% sparsity | Mamba-370M SSM · SSM module inference | speedup: 1.72x | Improved |
| Prune in_proj and out_proj as heavily as other FFN modules (ablation) | Mamba FFN pruning · Mamba architecture | degradation | Hurt |
| Prune SSM and FFN jointly with SparseSSM | full Mamba · multiple model scales | perplexity and zero-shot accuracy | Improved |
Closest papers by meaning
- Layer-Adaptive State Pruning for Deep State Space Models (NeurIPS 2024 · similarity 0.90)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.89)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.89)
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (ICML 2024 · similarity 0.89)
- Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · similarity 0.89)
- DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
