Research map / Architectures and efficiency / RNNs and normalization
Tuning Frequency Bias of State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes frequency bias of state space models via transfer functions, shows HiPPO initialization and conventional training keep a low-frequency bias, and proposes initialization scaling and a Sobolev-norm filter to tune it, reaching 88.26% average on LRA with S4D.
Problem
SSMs such as S4D are implicitly biased toward low-frequency content; it is unclear why this bias arises, whether training changes it, and how to tune it for tasks needing high-frequency information or long-range reasoning.
Approach
Formalizes frequency bias of SSMs via total variation of transfer functions, shows HiPPO initialization places eigenvalues' imaginary parts at low frequencies and gradients for those parameters are insensitive to high frequencies, and proposes two tuning knobs: scaling the imaginary parts of the initialization (alpha) and a Sobolev-norm filter (1+|s|)^beta applied to the transfer function (beta).
Key results
- The tuned S4D achieves an average of 88.26% on Long-Range Arena, versus 84.89% for S4D and 86.09% for S4 in the reported table.
- Image-denoising S4D models show the low-to-high pass-rate ratio decreasing from about 4.46e+07 (alpha=0.1, beta=-1) to about 5.96e-06 (alpha=100, beta=1), so higher alpha and beta strengthen high-frequency learning.
- In the sCIFAR-10 ablation, the best settings are alpha=4 with beta=0 and alpha=1 with beta=-0.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add initialization scaling and a Sobolev filter | S4D · LRA average | accuracy: 87.46 -> 88.26 | Helped |
| Add initialization scaling and a Sobolev filter | S4D · LRA Path-X | accuracy: 98.58 -> 97.84 | Hurt |
| Add initialization scaling and a Sobolev filter | S4D · LRA ListOps, Text, Retrieval, Image, and Pathfinder | accuracy | Helped |
| Increase initialization scale alpha | S4D image denoising · CelebA stripe-noise denoising | low-frequency to high-frequency pass-rate ratio | Helped |
| Increase Sobolev-filter beta | S4D image denoising · CelebA stripe-noise denoising | low-frequency to high-frequency pass-rate ratio | Helped |
| Increase alpha from the default while holding beta at 0 (ablation) | S4D · grayscale sCIFAR-10 | accuracy | Helped |
| Set alpha to 4 while holding beta at 0 (ablation) | S4D · grayscale sCIFAR-10 | accuracy | Helped |
| Decrease beta from the default while fixing alpha at 1 (ablation) | S4D · grayscale sCIFAR-10 | accuracy | Helped |
| Set beta to -0.5 while fixing alpha at 1 (ablation) | S4D · grayscale sCIFAR-10 | accuracy | Helped |
| Remove the skip connection D (ablation) | S4D · sCIFAR-10 | accuracy: less than 2% | Hurt |
| Remove A, B, and C (ablation) | S4D · sCIFAR-10 | accuracy | Hurt |
| Use conventional training with HiPPO initialization | S4D · sCIFAR-10 | relative change in Im(diag(A)): 1.43% | No clear effect |
Surprising findings
- Conventional training barely changes the imaginary parts of the S4D eigenvalues; they shift by only about 1.43% on average on sCIFAR-10.
- Increasing alpha helps high-frequency learning while decreasing beta also improves sCIFAR-10 results, so the two knobs act in different senses and can be combined.
Limitations
- Theoretical results are for a single SISO LTI system and the bounds hold asymptotically or under stated assumptions; the scaling rules give upper bounds only.
- The Sobolev filter theorem requires a growth condition on the functional derivative, and the authors note it can be implemented in practice even when the condition is violated.
Experimental setup
Datasets: CelebA sCIFAR-10 Long-Range Arena Moving MNIST
Benchmarks: Long-Range Arena sCIFAR-10 CelebA denoising
Metrics: test accuracy pass-rate ratio total variation
Baselines: S4 S4D DSS S4++ Regularized S4D Spectral SSM Liquid S4 S5
Builds on: S4D HiPPO S5 Neural tangent kernel spectral bias
Closest papers by meaning
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.86)
- Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning (NeurIPS 2024 · similarity 0.85)
- Elastic Spectral State Space Models for Train-Once Budgeted Inference (NeurIPS 2026 · similarity 0.85)
- HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.85)
- FreSh: Frequency Shifting for Accelerated Neural Representation Learning (ICLR 2025 · similarity 0.85)
- The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labels (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.