Research map / Architectures and efficiency / RNNs and normalization
Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Existing SSM initializations (HiPPO-based, S4D-Lin/Legs) assume exponential-decay measures and do not account for input temporal structure; the roles of timescale and state-matrix real/imaginary parts in SSM training are theoretically unclear.
Approach
placeholder
Key results
- On LRA, initializing 10% of real parts to zero raises S4D average accuracy from 84.89 to 87.08.
- Gram matrix for S4D-Lin is bounded as 0.2 < lambda_min(G) <= lambda_max(G) < sqrt(2), independent of hidden size m, whereas S4D-Real Hilbert-type Gram matrix condition number grows exponentially with m.
- On the ablation over sCIFAR with a 4-layer S4D, p=0.1 zero real-part initialization gives 84.60 accuracy vs 84.09 for p=0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize 10% of Re(w) to 0 | S4D · LRA ListOps | accuracy: 60.47 -> 61.44 | Helped |
| Initialize 10% of Re(w) to 0 | S4D · LRA Text | accuracy: 86.18 -> 88.05 | Helped |
| Initialize 10% of Re(w) to 0 | S4D · LRA Retrieval | accuracy: 89.46 -> 90.73 | Helped |
| Initialize 10% of Re(w) to 0 | S4D · LRA Image | accuracy: 88.19 -> 89.11 | Helped |
| Initialize 10% of Re(w) to 0 | S4D · LRA Pathfinder | accuracy: 93.06 -> 95.58 | Helped |
| Initialize 10% of Re(w) to 0 | S4D · LRA PathX | accuracy: 91.95 -> 97.55 | Helped |
| Initialize 10% of Re(w) to 0 | S4D · LRA average | accuracy: 84.89 -> 87.08 | Helped |
| Initialize 10% of Re(w) to 0 | diagonal SSM · i.i.d. copying task, sequence length 128 | task performance | Helped |
| Set Re(w) initialization to 0 rather than -0.5 | diagonal SSM · long-term memory task | test loss | Helped |
| Set Re(w) initialization to 0 rather than negative | diagonal SSM · decorrelated sMNIST | accuracy | Helped |
| Initialize fraction p=0.1 of Re(w) to 0 rather than p=0 (ablation) | 4-layer S4D · sCIFAR test | accuracy: 84.09 (0.47) -> 84.60 (0.38) | Helped |
| Increase zero-initialized Re(w) fraction to p=0.2 from p=0 (ablation) | 4-layer S4D · sCIFAR test | accuracy: 84.09 (0.47) -> 84.23 (0.49) | No clear effect |
Surprising findings
- Non-negative real parts persist after training even without reparameterization, yet performance still improves.
- The theoretical lower bound on lambda_min(G) is loose compared with the numerically observed minimal eigenvalue.
Limitations
- Analysis restricted to SISO diagonal SSMs with ZOH discretization and D=0; not directly applicable to bilinear discretization.
- Gram-matrix lower bound is not tight relative to numerical minimal eigenvalue.
- Real-part stability is not enforced during training, so Re(w) can become positive.
Experimental setup
Datasets: sequential MNIST sequential CIFAR10 Long Range Arena copying task (i.i.d. synthetic) synthetic Gaussian processes
Benchmarks: Long Range Arena sequential MNIST sequential CIFAR10
Metrics: test accuracy test loss expected output magnitude maximal eigenvalue of autocorrelation condition number
Baselines: S4D-Lin S4D-Legs S4D-Real S4D
Builds on: HiPPO S4 S4D
Closest papers by meaning
- From Generalization Analysis to Optimization Designs for State Space Models (ICML 2024 · similarity 0.91)
- HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.90)
- Uncovering the Spectral Bias in Diagonal State Space Models (NeurIPS 2025 · similarity 0.88)
- UnHiPPO: Uncertainty-aware Initialization for State Space Models (ICML 2025 · similarity 0.88)
- MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models (NeurIPS 2026 · similarity 0.87)
- Provable Benefits of Complex Parameterizations for Structured State Space Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.