Research map / Architectures and efficiency / RNNs and normalization
SpectraLDS: Provable Distillation for Linear Dynamical Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows how to distill a trained spectral transform unit (STU) layer into an explicit symmetric linear dynamical system via spectral-to-LDS conversion, enabling logarithmic-state recurrent inference with preserved accuracy.
Problem
Converting a trained Spectral Transform Unit (convolutional spectral filter layer) back into an explicit LDS recurrence with provable error bounds, to enable constant-time-per-token inference for symmetric LDSs with long memory.
Approach
Approximate each spectral filter as a linear combination of geometrically decaying LDS impulse responses sampled at random eigenvalues alpha (solved via least squares / OMP with a symmetric near-unit-radius sampling distribution), yielding a state-space realization whose state dimension is O(log L) with provable error bounds.
Key results
- Distilled SpectraLDS 340M reaches 39.03 average accuracy across nine language benchmarks versus 39.20 for Flash STU 340M, with state dimension 160.
- On synthetic LDS learning, SpectraLDS achieves MSE around 1e-5 to 1e-4 at sequence length 8192, outperforming gradient-descent LDS baselines that diverge at high state dimension.
- Reconstruction error of the spectral filters with an LDS of state dimension 160 is 7.69e-19 (MSE) for the 24 filters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill Flash STU convolutions into LDS layers | Flash STU 340M · Language benchmarks, Average | score: 39.20 -> 39.03 | No clear effect |
| Distill Flash STU convolutions into LDS layers | Flash STU 340M · Language benchmarks, Wino. | score: 51.85 -> 50.99 | No clear effect |
| Distill Flash STU convolutions into LDS layers | Flash STU 340M · Language benchmarks, ARC-e | score: 54.08 -> 54.17 | No clear effect |
| Train via STU and distill to an LDS instead of directly fitting LDS parameters | Synthetic symmetric LDS learning · Synthetic LDS, seq_len 8192, delta 1 × 10⁻², d_h 100 | Avg MSE: $2.15 \times 10^{-2}$ -> $3.35 \times 10^{-4}$ | Helped |
| Use symmetric near-unit-radius alpha sampling instead of uniform alpha in [-1, 1] (ablation) | Practical SpectraLDS reconstruction · Spectral-filter reconstruction | reconstruction error | Helped |
| Increase the number of alpha samples H while keeping state dimension small (ablation) | Practical SpectraLDS reconstruction · Spectral-filter reconstruction | reconstruction error | Hurt |
| Increase LDS state dimension h (ablation) | Practical SpectraLDS reconstruction · Spectral-filter reconstruction | reconstruction error | Helped |
| Replace STU spectral convolutions with recurrent LDS inference | STU layers · Autoregressive generation | runtime growth with sequence length | Helped |
Surprising findings
- Gradient-descent LDS baselines become unstable and often diverge at state dimension >= 1000 with learning rate 0.01, while the STU-based approach remains stable at learning rate 1.0.
- The condition number of the spectral coefficient matrix grows with h but lambda_max * h appears to stay bounded in experiments, contrary to the worst-case exponential bound.
Limitations
- Theoretical bound depends on lambda_max of the pseudo-inverse, which is not provably bounded and can be exponentially large for h near k.
- LDS layers require float64 precision; unclear whether stable below float64.
- Restricted to symmetric A (real eigenvalues); STU-T tensor-dot approximation does not inherit the same guarantees.
Experimental setup
Datasets: synthetic symmetric LDS sequences
Benchmarks: MMLU HellaSwag PIQA BoolQ WinoGrande CSQA OBQA ARC-e ARC-c
Metrics: accuracy average accuracy MSE runtime
Baselines: Flash STU Transformer Ho-Kalman LDS gradient descent Epoched Future Fill Naive convolution
Builds on: Spectral Transform Unit spectral filtering Flash-STU Laughing Hyena future fill
Closest papers by meaning
- ResKoopNet: Learning Koopman Representations for Complex Dynamics with Spectral Residuals (ICML 2025 · similarity 0.86)
- Universal Sequence Preconditioning (NeurIPS 2025 · similarity 0.86)
- The Power of Second Order Methods for Sequence Preconditioning (NeurIPS 2026 · similarity 0.85)
- Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction (NeurIPS 2024 · similarity 0.85)
- Deep-Koopman-KANDy: Dictionary Discovery for Deep-Koopman Operators with Kolmogorov-Arnold Networks for Dynamics (NeurIPS 2026 · similarity 0.85)
- A Memory Efficient Unified Algorithm for Online Learning of Linear Dynamical Systems (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.