Research map / Architectures and efficiency / RNNs and normalization
Hankel Singular Value Regularization for Highly Compressible State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Regularizes the nuclear norm of Hankel singular values during training of S5-style rotation-parametrized SSM layers so they can be compressed via balanced truncation with little accuracy loss.
Problem
SSM layers trained with standard optimization have slowly decaying Hankel singular values, so they cannot be compressed to lower state order without large accuracy loss.
Approach
Add a differentiable Hankel nuclear norm regularizer during training, enabled by a block-rotation SSM parametrization whose Lyapunov equations decompose into 2x2 blocks, giving O(n^2) gramian computation; compress afterward with balanced truncation and layer-adaptive rank selection.
Key results
- At 80% truncation on sMNIST, HSVR retains 98.90% test accuracy versus about 11% for unregularized training.
- On IMDB, HSVR keeps about 86.4% accuracy at 90% truncation, while unregularized training drops to 50%.
- Blocked gramian computation adds only 1.12x to 1.59x training time per epoch, versus 27.3x for the naive Lyapunov solver on sMNIST.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Hankel singular value regularization during training | SSM · sCIFAR (grayscale), 80% truncation | test accuracy: 21.14 -> 81.37 | Helped |
| add Hankel singular value regularization during training | SSM · sMNIST, 80% truncation | test accuracy: 11.05 -> 98.90 | Helped |
| add Hankel singular value regularization during training | SSM · IMDB, 80% truncation | test accuracy: 51.32 -> 86.97 | Helped |
| add Hankel singular value regularization during training | SSM · PATH, 80% truncation | test accuracy: 50.35 -> 50.50 | No clear effect |
| add Hankel singular value regularization during training | SSM · PATH-X, 80% truncation | test accuracy: 50.16 -> 54.02 | Helped |
| use Hankel singular value regularization instead of L1 regularization | SSM · sCIFAR (grayscale), 80% truncation | test accuracy: 69.90 -> 81.37 | Helped |
| use Hankel singular value regularization instead of L1 regularization | SSM · IMDB, 80% truncation | test accuracy: 50.16 -> 86.97 | Helped |
| use Hankel singular value regularization instead of L1 regularization | SSM · sMNIST, 80% truncation | test accuracy: 90.12 -> 98.90 | Helped |
| replace block-wise Lyapunov solver with naive Lyapunov solver for HSVR | SSM training · sMNIST | relative training time: 1.12× -> 27.3× | Hurt |
| compress the trained SSM at 80% truncation | HSVR sCIFAR model · sCIFAR (grayscale) | runtime ratio: 0.45 | Helped |
Surprising findings
- On PATH, HSVR gave little difference in the Hankel singular value distribution compared with unregularized training, and the smallest accuracy gain.
Limitations
- Models must be trained with the regularizer; not applicable to pretrained models.
- Compression restricted to linear methods (balanced truncation).
- Only linear time-invariant SSMs are considered; time-varying extension remains future work.
Experimental setup
Datasets: CIFAR-10 grayscale (sCIFAR) MNIST (sMNIST) IMDB PATH PATH-X
Benchmarks: sCIFAR (grayscale) sMNIST IMDB PATH PATH-X
Metrics: test accuracy truncation ratio inference runtime ratio relative training time
Baselines: LAST global pruning uniform pruning no regularization L1 regularization of diagonal blocks
Builds on: S5 HiPPO balanced truncation Bartels-Stewart algorithm Long Range Arena
Closest papers by meaning
- HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.88)
- From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics (ICLR 2025 · similarity 0.84)
- Layer-Adaptive State Pruning for Deep State Space Models (NeurIPS 2024 · similarity 0.84)
- Cauchy-Schwarz Regularizers (ICLR 2025 · similarity 0.84)
- Tuning Frequency Bias of State Space Models (ICLR 2025 · similarity 0.84)
- Provable Benefits of Complex Parameterizations for Structured State Space Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.