Research map / Architectures and efficiency / RNNs and normalization
HOPE for a Robust Parameterization of Long-memory State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOPE, an SSM parameterization of LTI systems by Markov parameters of a discrete Hankel operator (sampled nonuniformly in the transfer function), arguing it avoids the rank loss, perturbation instability, and memory decay of canonical (A,B,C) SSMs.
Problem
SSMs with LTI systems need careful initialization and small learning rates for the state matrices because their Hankel singular values decay fast, causing loss of expressiveness and numerical instability during training, plus exponentially decaying memory.
Approach
Parameterize the LTI system directly by the Markov parameters h of a discrete Hankel matrix (n complex parameters instead of 3n for S4D), discretize with a trainable step size by nonuniformly sampling the transfer function via FFT; random Hankel matrices are almost surely high-rank, perturbation bounds are Lipschitz in h, and the impulse response does not decay within a window of length n.
Key results
- HOPE-SSM achieves 87.85 average accuracy on Long-Range Arena, versus 86.09 for S4 and 84.89 for S4D, and best on Text, Retrieval, and Pathfinder.
- On noise-padded sCIFAR-10, HOPE-SSM significantly outperforms S4D (table reported in paper figure).
- Impulse response of trained HOPE-SSM does not decay over t in [0,64] while the trained S4D decays exponentially.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace S4D LTI blocks with Hankel-parameterized HOPE blocks | S4D · LRA ListOps | accuracy: 62.75 -> 62.60 | Hurt |
| Replace S4D LTI blocks with Hankel-parameterized HOPE blocks | S4D · LRA Text | accuracy: 89.60 -> 89.83 | Helped |
| Replace S4D LTI blocks with Hankel-parameterized HOPE blocks | S4D · LRA Retrieval | accuracy: 91.25 -> 91.80 | Helped |
| Replace S4D LTI blocks with Hankel-parameterized HOPE blocks | S4D · LRA Image | accuracy: 89.50 -> 88.68 | Hurt |
| Replace S4D LTI blocks with Hankel-parameterized HOPE blocks | S4D · LRA Pathfinder | accuracy: 95.60 -> 95.73 | Helped |
| Replace S4D LTI blocks with Hankel-parameterized HOPE blocks | S4D · LRA Path-X | accuracy: 98.58 -> 98.45 | Hurt |
| Replace S4D LTI blocks with Hankel-parameterized HOPE blocks | S4D · LRA average | accuracy: 87.46 -> 87.85 | Helped |
| Use HOPE instead of S4D for the LTI parameterization | S4D · Noise-padded sCIFAR-10 | test accuracy | Helped |
| Use Hankel Markov parameters to parameterize the LTI system | S4D · LTI parameter count | parameter count: 1/3 | Helped |
| Use HOPE parameterization rather than S4D | S4D · Impulse response over t ∈ [0,64] | memory decay | Helped |
Surprising findings
- Training S4D LTI systems with init_1 (high initial rank) reduces Hankel singular values rapidly, losing expressiveness, so training the LTI systems can impair performance.
Limitations
- Memory non-decay holds only within a window t in [0, n]; beyond that the impulse response is zero, requiring a small sampling period for longer dependencies.
- Results on LRA are reported against S4/S4D with same model sizes; the theory relies on random-initialization assumptions and asymptotic arguments.
Experimental setup
Datasets: sCIFAR-10 Long-Range Arena noise-padded sCIFAR-10
Benchmarks: Long-Range Arena sCIFAR-10 noise-padded sCIFAR-10
Metrics: test accuracy median and std over 5 seeds Hankel singular values impulse response decay
Baselines: S4 S4D S5 Liquid S4 Reg. S4D Spectral SSM DSS S4++
Builds on: S4 S4D HiPPO S5
Closest papers by meaning
- Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models (ICLR 2025 · similarity 0.90)
- MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models (NeurIPS 2026 · similarity 0.89)
- StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization (ICML 2024 · similarity 0.88)
- Hankel Singular Value Regularization for Highly Compressible State Space Models (NeurIPS 2025 · similarity 0.88)
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.87)
- Elastic Spectral State Space Models for Train-Once Budgeted Inference (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.