Research map / Architectures and efficiency / Pruning and efficient tuning
Layer-Adaptive State Pruning for Deep State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAST, a layer-adaptive structured state-pruning method for diagonal SSMs (S4D, S5) that scores states by H-infinity subsystem norms with layer-wise energy normalization, enabling one-shot pruning without retraining.
Problem
Deep state space models use high state dimensions that are costly; existing SSMs lack state-dimension optimization methods, and pruning SSM states requires criteria suited to their transfer functions and nonlinear stacked layers.
Approach
Extends single-system modal truncation (H-infinity norm of subsystems) to multi-layer nonlinear SSMs by normalizing each state's H-infinity score by the sum of remaining scores in its layer (energy normalization), yielding global cross-layer pruning scores that upper-bound model-level output energy loss.
Key results
- On average, LAST prunes 33% of states in S5 (MIMO) with 0.52% accuracy loss without retraining, versus 4.32% for Uniform H-infinity and 7.51% for Global H-infinity.
- For S4D (multi-SISO), LAST prunes 25% of states with 0.32% average accuracy loss, compared with 0.39% for Uniform and 1.16% for Global H-infinity.
- Unstructured random pruning at 33% ratio causes 59.92% average accuracy loss in S5, showing the importance of state-level structured pruning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LAST instead of Uniform H∞ pruning | S4D · All tasks, compressible cases excluded | accuracy loss: 0.52 -> 0.42 | Helped |
| Add energy normalization to global H∞ pruning (ablation) | S4D · All tasks, compressible cases excluded | accuracy loss: 1.55 -> 0.42 | Helped |
| Use LAST instead of Uniform H∞ pruning | S5 · All tasks, compressible cases excluded | accuracy loss: 4.80 -> 0.58 | Helped |
| Add energy normalization to global H∞ pruning (ablation) | S5 · All tasks, compressible cases excluded | accuracy loss: 8.35 -> 0.58 | Helped |
| Use LAST instead of Uniform H∞ pruning at 30% pruning | S5 · Pathfinder | accuracy: 71.38 -> 94.45 | Helped |
| Add energy normalization to global H∞ pruning at 30% pruning (ablation) | S5 · Pathfinder | accuracy: 57.20 -> 94.45 | Helped |
| Use LAST instead of Uniform H∞ pruning at 30% pruning | S5 · Path-X | accuracy: 90.90 -> 97.95 | Helped |
| Add energy normalization to global H∞ pruning at 30% pruning (ablation) | S5 · Path-X | accuracy: 69.21 -> 97.95 | Helped |
| Prune whole states rather than randomly selected individual matrix elements | S5 · All tasks, compressible cases excluded | accuracy loss: 66.58 -> 32.82 | Helped |
Surprising findings
- Unstructured random pruning of S5 at a similar ratio is far worse than structured random state pruning, with average accuracy loss of 59.92% versus 29.53%.
- Global H-infinity without energy normalization can be worse than uniform layer-wise pruning, showing that naive cross-layer score comparison fails.
Limitations
- When and how often to prune during training remains unresolved.
- Evaluated only on a specific set of tasks; adaptation to other tasks is untested.
- Pruning for S4D is implemented by masking rather than physically removing states.
Experimental setup
Datasets: Long Range Arena Speech Commands sMNIST psMNIST CIFAR-10
Benchmarks: Long Range Arena Speech Commands sMNIST psMNIST sCIFAR
Metrics: accuracy accuracy loss pruning ratio
Baselines: Uniform H-infinity Global H-infinity Random state pruning Unstructured random pruning Uniform Magnitude Global Magnitude LAMP
Builds on: S4D S5 modal truncation LAMP
Closest papers by meaning
- Exploring Token Pruning in Vision State Space Models (NeurIPS 2024 · similarity 0.88)
- From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics (ICLR 2025 · similarity 0.87)
- State Space Models are Provably Comparable to Transformers in Dynamic Token Selection (ICLR 2025 · similarity 0.86)
- Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning (NeurIPS 2025 · similarity 0.86)
- Exploring Adversarial Robustness of Deep State Space Models (NeurIPS 2024 · similarity 0.85)
- On Feature Learning in Structured State Space Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.