Research map / Architectures and efficiency / RNNs and normalization
Uncovering the Spectral Bias in Diagonal State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes diagonal SSM initialization from a frequency perspective and proposes S4D-DFouT, which places poles uniformly on the discrete unit circle, achieving strong LRA results and training from scratch on PathX-256.
Problem
Diagonal state space model (SSM) initializations (S4D-Lin, S4D-Inv) are defined in continuous time and discretized with a step Delta, which entangles decay and frequency and makes performance sensitive to Delta; the role of diagonal initializations is theoretically unexplained.
Approach
Initialize diagonal SSM poles directly in the discrete domain on uniformly spaced angles (DFT frequencies) with a shared learnable damping xi, synchronized with phase offsets across the SSMs of a layer, so frequency coverage is alias-free and independent of Delta.
Key results
- S4D-DFouT reaches 87.89% on LRA PathX-256 (65,536 length) trained from scratch, the first such result without self-pretraining.
- S4D-DFouT achieves 61.89 on ListOps and 94.30 on Pathfinder, the best diagonal variant on those tasks in the LRA table.
- On Speech Commands 8kHz zero-shot, S4D-DFouT reaches 93.85% versus 91.80% for the best prior diagonal baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use S4D-DFouT instead of S4 + Self-Pretrain | S4 state-space model · LRA PathX-256 | accuracy: 87.11 -> 87.89 | Helped |
| use S4D-DFouT instead of S4D-Lin | diagonal SSM · Speech Commands 8kHz | accuracy: 91.58 (0.33) -> 93.85 (0.27) | Helped |
| use S4D-DFouT instead of S4D-Lin | diagonal SSM · Speech Commands 16kHz | accuracy: 96.25 (0.03) -> 96.07 (0.19) | Hurt |
| replace uniform DFouT frequencies with token-period frequencies (ablation) | S4D-DFouT · Speech Commands 8kHz | accuracy: 93.85 (0.27) -> 92.92 (0.22) | Hurt |
| replace uniform DFouT frequencies with random frequencies (ablation) | S4D-DFouT · psCIFAR | accuracy: 65.72 -> 60.55 | Hurt |
| replace uniform DFouT frequencies with random frequencies (ablation) | S4D-DFouT · Speech Commands 16kHz | accuracy: 96.07 (0.19) -> 95.43 (0.26) | Hurt |
| use contiguous-batch pole synchronization instead of DFouT synchronization (ablation) | S4D-DFouT · Speech Commands 8kHz | accuracy: 93.85 (0.27) -> 0.07 (0.14) | Hurt |
| use contiguous-batch pole synchronization instead of DFouT synchronization (ablation) | S4D-DFouT · psCIFAR | accuracy: 65.72 -> 63.61 | Hurt |
| increase the upper bound on the decay parameter (ablation) | S4D-DFouT and baseline diagonal SSMs · LRA Image and Pathfinder | accuracy | Mixed |
| reduce kernel length to 32 coefficients (ablation) | S4D-Lin on sCIFAR · sCIFAR | accuracy | No clear effect |
| replace subsequent SSM layers with identity mappings, keeping only the first k layers active (ablation) | S4D-Lin on sCIFAR · sCIFAR | accuracy | Hurt |
| use S4D-DFouT instead of S4D-Inv | diagonal SSM · sCIFAR | accuracy: 90.69 (0.06) -> 89.87 (0.07) | Hurt |
Surprising findings
- Learned SSM kernels on sCIFAR show a local-attention profile, and kernels of length 32 retain original accuracy, suggesting LRA tasks can be solved with short receptive fields.
- S4D-DFouT does not help on permuted sCIFAR (65.7%), where local structure is removed.
Limitations
- Gains over prior diagonal methods are small on several LRA tasks (e.g., Text and Retrieval trail S4D-Lin/S5), and PathX-128 underperforms S4D-Lin with tuned Delta.
- Not evaluated on textual data or LLMs; psCIFAR not improved.
- Authors note LRA difficulty may be overestimated, calling for new benchmarks.
Experimental setup
Datasets: LRA sCIFAR psCIFAR Speech Commands BIDMC continuous copying (synthetic)
Benchmarks: Long Range Arena sCIFAR psCIFAR Speech Commands BIDMC
Metrics: accuracy RMSE reconstruction MSE
Baselines: S4-LegS S4-FouT S5 S4D-Lin S4D-Inv S4D-LegS S4 + Self-Pretrain LRU Liquid-S4 MEGA
Builds on: S4 S4D HiPPO DSS S5
Closest papers by meaning
- Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models (ICLR 2025 · similarity 0.88)
- Provable Benefits of Complex Parameterizations for Structured State Space Models (NeurIPS 2024 · similarity 0.85)
- HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.85)
- Spectral Phase Transition and Optimal PCA in Block-Structured Spiked Models (ICML 2024 · similarity 0.84)
- Tuning Frequency Bias of State Space Models (ICLR 2025 · similarity 0.84)
- Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.