Research map / Architectures and efficiency / RNNs and normalization
Fixed-Point RNNs: Interpolating from Diagonal to Dense
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fixed-Point RNNs, which implicitly parameterize dense linear RNNs as fixed points of diagonal linear RNN iterations, trading sequential compute for expressivity, and an FP-Mamba variant evaluated on copying and state-tracking (A5, S5).
Problem
Diagonal (channel-wise) linear RNNs and SSMs such as Mamba lack full state-tracking expressivity, while dense input-dependent linear RNNs are hard to parallelize and stabilize.
Approach
Parameterize a dense linear RNN implicitly as the fixed point of a diagonal linear RNN, with contractive input normalization and channel mixers (DPLR, Householder, Kronecker) so that fixed-point iteration converges; the number of iterations adaptively trades parallelism for expressivity, and FP-Mamba adds matrix states and hidden-state dependence.
Key results
- FP-Mamba with Kronecker or Householder channel mixers achieves length generalization on A5 beyond the training length of 16, while Mamba and Mamba-2 fail to learn A5 even at training length.
- On copying, FP-Mamba closes the length-generalization gap to Gated DeltaNet at x2 length, where Mamba 1 and 2 struggle.
- The effective number of fixed-point iterations adapts to task difficulty, staying well below sequence length on copying and language pretraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FP-Mamba with a Kronecker channel mixer | FP-Mamba · A5 and S5 state-tracking | state-tracking performance | Helped |
| use a Kronecker channel mixer instead of Householder or DPLR mixers (ablation) | FP-Mamba · A5 state-tracking | sequence length generalization | Helped |
| add shifted hidden-state dependence to b and c (ablation) | FP-Mamba · copy task | copying ability | Helped |
| add shifted hidden-state dependence to lambda and Q as well as b and c (ablation) | FP-Mamba · copy task | length generalization | Helped |
| use FP-Mamba rather than Mamba or Mamba-2 | sequence models · copy task | length generalization | Helped |
| cap fixed-point iterations below the training sequence length (ablation) | FP-Mamba-H · A5 state-tracking | state-tracking performance | Hurt |
| randomize the maximum number of fixed-point iterations per batch (ablation) | FP-Mamba-H · A5 state-tracking | training efficiency | Helped |
| use LSTM rather than Mamba or Mamba-2 | sequence models · A5 and S5 state-tracking | state-tracking capability | Helped |
Surprising findings
- Gradients computed only at the fixed point (no truncated backpropagation through iterations) suffice to stabilize training, adding no backward-pass overhead.
- Mamba and Mamba-2 cannot learn A5 even at the training sequence length.
Limitations
- Fixed-point iterations add sequential forward overhead and worst case O(T^2) runtime if iterations scale with sequence length.
- Experiments are limited to toy tasks (copying, A5/S5, modular arithmetic) plus small-scale language pretraining; no efficient fused kernel yet.
- Convergence and uniqueness of fixed points are not guaranteed with hidden-state dependence.
Experimental setup
Datasets: A5 state tracking S5 state tracking copying task modular arithmetic FineWeb
Benchmarks: A5 S5 copying task FineWeb language modeling
Metrics: sequence length generalization accuracy validation perplexity
Baselines: Mamba Mamba-2 Gated DeltaNet LSTM DeltaProduct
Builds on: Mamba DeltaNet Deep Equilibrium Models Gated DeltaNet DeltaProduct
Closest papers by meaning
- Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues (ICLR 2025 · similarity 0.87)
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.87)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.85)
- Do Transformer Interpretability Methods Transfer to RNNs? (AAAI 2025 · similarity 0.85)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.85)
- Improving Bilinear RNN with Closed-loop Control (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.