Research map / Architectures and efficiency / RNNs and normalization
Robustifying State-space Models for Long Sequences via Approximate Diagonalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes perturb-then-diagonalize (PTD), a backward-stable approximate diagonalization of the HiPPO matrix, yielding S4-PTD and S5-PTD SSMs that are robust to Fourier-mode input perturbations and match or beat S4D/S5 on Long-Range Arena.
Problem
Diagonalizing the HiPPO state matrix is exponentially ill-conditioned, so diagonal SSMs (S4D, S5) replace it with a different well-conditioned matrix, which makes them non-robust to certain Fourier-mode input perturbations.
Approach
Instead of diagonalizing HiPPO exactly, add a small optimized perturbation E to the HiPPO matrix and diagonalize A_H+E (a backward-stable eigensolver view), so the diagonal model's transfer function approximates the DPLR HiPPO transfer function uniformly, with a soft-constrained optimization trading off eigenvector condition number against perturbation size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace S4D initialization with perturb-then-diagonalize HiPPO initialization | S4D · Long-Range Arena (Avg.) | accuracy: 84.89 -> 86.58 | Helped |
| Replace S4 initialization with perturb-then-diagonalize HiPPO initialization | S4 · Long-Range Arena (Avg.) | accuracy: 86.09 -> 86.58 | Helped |
| Replace S5 initialization with perturb-then-diagonalize HiPPO initialization | S5 · Long-Range Arena (Avg.) | accuracy: 87.46 -> 87.61 | Helped |
| Vary perturbation size relative to the HiPPO matrix (ablation) | S4-PTD · sCIFAR | test accuracy | Mixed |
| Use perturb-then-diagonalize initialization instead of S4D initialization | S4D · sCIFAR with sinusoidal noise near transfer-function spikes | test accuracy | Helped |
Surprising findings
- The S4D model fails on sinusoidal inputs near its transfer-function spike around frequency 83, with predicted amplitude going negative (about -4) when the true amplitude is in [0,1].
- The paper proves that the S4D and S4 transfer functions do not converge uniformly on the imaginary axis even though outputs converge for fixed smooth inputs.
Limitations
- Perturbation size must be tuned (ratio to HiPPO norm between 1e-2 and 1 works best); accuracy drops when it is too small or too large.
- Robustness evaluated mainly against worst-case noise crafted to break S4D; average-case benefits are less clear.
- The S5-PTD model does not outperform S5 on all tasks (e.g., Image and Path-X are slightly lower or comparable).
Experimental setup
Datasets: Long-Range Arena sequential CIFAR-10 CIFAR-10 synthetic sinusoidal signals
Benchmarks: Long-Range Arena sequential CIFAR-10
Metrics: test accuracy eigenvector condition number interpolation/extrapolation error
Baselines: S4 S4D S5 Liquid-S4 Transformer Luna-256 H-Transformer-1D CCNN
Builds on: S4 S4D S5 HiPPO
Closest papers by meaning
- Uncovering the Spectral Bias in Diagonal State Space Models (NeurIPS 2025 · similarity 0.90)
- HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.88)
- Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models (ICLR 2025 · similarity 0.87)
- Provable Benefits of Complex Parameterizations for Structured State Space Models (NeurIPS 2024 · similarity 0.87)
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.86)
- Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.