Research map / Architectures and efficiency / Attention and transformers
State-Free Inference of State-Space Models: The *Transfer Function* Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rational Transfer Function (RTF) parametrization of SSMs that computes the convolution kernel spectrum via a single FFT, giving state-free parallel inference with O(l) memory and O(l log l) time.
Problem
Parallel (sequence-parallel) inference for state-space models with large state sizes is memory-intensive (scan-based methods scale as O(ln)) or slow and platform-dependent (S4/S4D Cauchy/Vandermonde kernels at O((l+n)log^2(l+n))).
Approach
Parametrize the SSM directly by its rational transfer function coefficients (a, b) rather than matrices A,B,C; truncated transfer function evaluated at roots of unity via one FFT gives the convolution kernel spectrum, so the state size is absorbed into sequence length without extra memory. Also proposes a zero initialization for stability instead of Montel's constraint.
Key results
- On Long Range Arena, RTF averages 87.71 versus S5 87.46 and S4 86.69, and is state-of-the-art among attention-free models on Retrieval (92.04) and Image (90.51).
- Hyena-RTF reaches 18.0 perplexity on WikiText-103 versus 18.5 for Hyena and 18.3 for Hyena-S5.
- RTF training speed is reported as about 35% faster than S4 layers on LRA on average (per abstract); RTF memory and latency stay flat as state size grows while S5 memory grows with state size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Hyena's MLP-parameterized filters with RTF | Hyena · WikiText-103 | perplexity: 18.5 -> 18.0 | Helped |
| Increase state size from 64 to 128 | RTF · Copying | accuracy: 22.1 -> 93.3 | Helped |
| Increase state size from 64 to 128 | S4 · Copying | accuracy: 29.3 -> 34.2 | Helped |
| Increase state size from 64 to 1024 | RTF · Copying | accuracy: 22.1 -> 100 | Helped |
| Use impulse initialization instead of Xavier initialization with Montel constraint (ablation) | RTF · WikiText-103 (25 epochs) | perplexity: 26.512 -> 26.093 | Helped |
| Remove Montel constraint while retaining Xavier initialization (ablation) | RTF · LRA Image | accuracy: 89.2 -> 90.0 | Helped |
| Use impulse initialization instead of Xavier initialization with Montel constraint (ablation) | RTF · LRA Image | accuracy: 89.2 -> 90.1 | Helped |
| Use RTF instead of LH at state size 4 | Hyena filter distillation · Distillation | MSE: 0.032 -> 0.018 | Helped |
| Use RTF instead of LH at state size 16 | Hyena filter distillation · Distillation | MSE: 0.009 -> 0.013 | Hurt |
| Use RTF instead of LH at state size 64 | Hyena filter distillation · Distillation | MSE: 0.007 -> 0.011 | Hurt |
Surprising findings
- Modal (diagonal) distillation via Laughing Hyena gave lower distillation MSE than RTF at state sizes 16 and 64, despite RTF's theoretical expressivity advantage.
- Montel's stability constraint was found to over-constrain the SSM and hurt performance; zero initialization worked better.
Limitations
- Path-X failed to learn at state size 64 and required increasing state size to 2048
- Distillation results show the modal representation can outperform RTF at some state sizes
- Language modeling experiments are small-scale (125M-160M) and the authors call for scaling
Experimental setup
Datasets: Long Range Arena WikiText-103 The Pile Copying synthetic task Delay synthetic task
Benchmarks: Long Range Arena WikiText-103 LM-Evaluation-Harness (Winogrande, PIQA, HellaSwag, OpenbookQA) Copying Delay
Metrics: accuracy perplexity inference latency GPU memory MSE RMSE
Baselines: S4 S4D S5 SpaceTime LRU Hyena Laughing Hyena Transformer H3 Mega
Builds on: S4 S4D SpaceTime Hyena Laughing Hyena
Closest papers by meaning
- Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory (ICLR 2025 · similarity 0.86)
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.85)
- State Space Models are Provably Comparable to Transformers in Dynamic Token Selection (ICLR 2025 · similarity 0.85)
- Elastic Spectral State Space Models for Train-Once Budgeted Inference (NeurIPS 2026 · similarity 0.85)
- Tuning Frequency Bias of State Space Models (ICLR 2025 · similarity 0.84)
- B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.