Research map / Probabilistic and scientific modeling / Dynamical systems
UnHiPPO: Uncertainty-aware Initialization for State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends HiPPO to noisy measurements by casting it as a linear stochastic control/Kalman filtering problem, yielding UnHiPPO initialization for state space models (LSSL) that filters measurement noise at no runtime cost.
Problem
HiPPO-initialized state space models assume noise-free input signals, which is violated in practice, making them fragile under measurement noise.
Approach
Reinterpret HiPPO as a linear stochastic control problem where the signal is a noise-free control input, then reformulate so the signal becomes noisy observations of a latent state; the resulting Kalman filter posterior mean is a linear recurrence (UnHiPPO matrices) used as an SSM initialization.
Key results
- UnLSSL improves robustness to noise levels different from the training noise level, both lower and higher, on speech classification.
- With training noise, UnHiPPO slightly hurts accuracy at low noise but improves it as noise increases.
- On SC10 with rho=0.1 noise, closed-form discretization reaches 79.3% accuracy versus 79.8% for backward Euler.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace HiPPO initialization with UnHiPPO initialization | LSSL sequence classifier · FSD and SC10, evaluation noise levels different from training noise | accuracy | Helped |
| Replace HiPPO initialization with UnHiPPO initialization | LSSL sequence classifier · FSD and SC10, training and evaluation at the same noise level | accuracy | Mixed |
| Tune σ² to the expected data noise level | UnLSSL · SC10, ρ = 10^-1 | fit on noisy data | Helped |
| Use backward Euler instead of closed-form discretization (ablation) | UnLSSL · SC10, ρ = 10^-1 noise | accuracy: 79.3% vs 79.8% | No clear effect |
| Use trapezoidal discretization instead of the closed-form solution (ablation) | UnLSSL · SC10 long sequences | stability | Hurt |
| Use forward Euler discretization instead of the closed-form solution (ablation) | UnLSSL · SC10 long sequences | stability | Hurt |
Surprising findings
- Trapezoidal and forward Euler discretizations of the UnHiPPO matrix produce diverging SSMs on long SC10 sequences.
Limitations
- Structured (diagonal plus low-rank) parametrization not found for the regularized HiPPO or UnHiPPO matrices, so O(N^2) cost remains.
- sigma^2 hyperparameter has large scale and cannot be interpreted directly as noise variance.
- SSM parameters are kept fixed in experiments rather than fine-tuned.
Experimental setup
Datasets: Free Spoken Digits (FSD) Speech Commands (SC10)
Benchmarks: Free Spoken Digits SC10
Metrics: classification accuracy accuracy difference between UnLSSL and LSSL
Baselines: LSSL HiPPO initialization
Builds on: HiPPO LSSL S4 Kalman filter
Closest papers by meaning
- Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models (ICLR 2025 · similarity 0.88)
- HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.86)
- MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models (NeurIPS 2026 · similarity 0.85)
- Predictability Enables Parallelization of Nonlinear State Space Models (NeurIPS 2025 · similarity 0.85)
- Gated Inference Network: Inference and Learning State-Space Models (NeurIPS 2024 · similarity 0.85)
- From Generalization Analysis to Optimization Designs for State Space Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.