Research map / Probabilistic and scientific modeling / Dynamical systems
Provable Benefits of Complex Parameterizations for Structured State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically proves that complex diagonal SSMs can express oscillatory LTI mappings that real diagonal SSMs need exponentially large dimension or parameters for, and validates this with controlled experiments and Mamba selectivity ablations.
Problem
Explaining theoretically why complex parameterizations benefit structured state space models (diagonal SSMs such as S4 and Mamba), where real and complex SSMs are both universal in expressiveness.
Approach
Establishes formal expressiveness and practical-learnability separations between real and complex diagonal SSMs: real SSMs need dimension or parameter magnitude exponential in the time horizon to approximate oscillatory or generic LTI mappings, while complex SSMs need only linear-in-t dimension and parameters.
Key results
- Real SSM trained with Adam/AdamW/RAdam at t=32 retains normalized approximation error of about 0.54 to 0.81 across copy, random and oscillatory mappings, while complex SSM reaches errors around 1e-5 to 1e-3 at t up to 256.
- S4 on sequential CIFAR-10: complex parameterization reached 89.10% test accuracy versus at most 78.27% for real parameterization.
- Mamba with selectivity: on copy, complex reached at least 93.05% vs at most 80.17% for real; on induction-head with selective B and C, real (97.35 to 98.3%) matched or beat complex (93.93 to 97.64%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use complex instead of real parameterization | Analyzed SSM · Copy mapping, t=32 | approximation error: 0.767 -> 1.6 \times 10^{-5} | Helped |
| Use complex instead of real parameterization | Analyzed SSM · Random mapping, t=32 | approximation error: 0.536 -> 6.3 \times 10^{-5} | Helped |
| Use complex instead of real parameterization | Analyzed SSM · Oscillatory mapping, t=32 | approximation error: 0.812 -> 1.6 \times 10^{-4} | Helped |
| Use complex instead of real parameterization | S4 · Sequential CIFAR-10 test | accuracy: 78.27\% -> 89.10\% | Helped |
| Use complex instead of real parameterization | Mamba · Copy task | accuracy: 80.17\% -> 93.05\% | Helped |
| Use complex instead of real parameterization | Mamba · Induction-head task with selective B, C, and Δ | accuracy: 97.35\% to 98.3\% vs. 93.93\% to 97.64\% | Mixed |
| Make Δ non-selective while retaining selective B and C (ablation) | Mamba with real parameterization · Induction-head task | accuracy: 97.35 to 98.3 -> 97.9 to 98.62 | No clear effect |
| Make Δ non-selective while retaining selective B and C (ablation) | Mamba with complex parameterization · Induction-head task | accuracy: 93.93 to 97.64 -> 90.18 to 95.86 | Hurt |
| Make C non-selective (ablation) | Mamba with real parameterization · Induction-head task, selective B and Δ | accuracy: 97.35 to 98.3 -> 61.82 to 71.28 | Hurt |
| Make C non-selective (ablation) | Mamba with complex parameterization · Induction-head task, selective B and Δ | accuracy: 93.93 to 97.64 -> 91.93 to 96.77 | No clear effect |
| Make B non-selective (ablation) | Mamba · Induction-head task, selective C and Δ | accuracy: Real: 56.78 to 69.54; complex: 92.52 to 96.91 | Mixed |
| Make B and C non-selective (ablation) | Mamba · Induction-head task, selective Δ | accuracy: Real: 15.48 to 26.33; complex: 68.54 to 79.71 | Mixed |
Surprising findings
- With selectivity on both B and C, real parameterization matches or outperforms complex on the induction-head task, suggesting selectivity may close the gap.
- Complex SSMs need only dimension and parameters linear in t, while real SSMs need exponential dimension or parameter magnitude for oscillatory or generic mappings.
Limitations
- Does not quantify how prevalent the expressiveness separation is across mappings.
- Practical learnability of complex SSMs is supported only empirically, not formally proven.
- Experiments include only a single real-world setting (sequential CIFAR-10).
Experimental setup
Datasets: sequential CIFAR-10 copy task induction-head task synthetic LTI mappings (copy, random, oscillatory)
Benchmarks: sequential CIFAR-10 (Long Range Arena) copy task induction-head task
Metrics: approximation error test accuracy
Baselines: real-parameterized SSM S4 with real parameterization Mamba with real parameterization
Builds on: S4 Mamba LRU S5
Closest papers by meaning
- On Feature Learning in Structured State Space Models (NeurIPS 2024 · similarity 0.88)
- State Space Models are Provably Comparable to Transformers in Dynamic Token Selection (ICLR 2025 · similarity 0.87)
- Theoretical Foundations of Deep Selective State-Space Models (NeurIPS 2024 · similarity 0.87)
- HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.87)
- Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models (ICLR 2025 · similarity 0.87)
- From Generalization Analysis to Optimization Designs for State Space Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.