Research map / Architectures and efficiency / RNNs and normalization

Linear approximations to HMM filtering

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: LSTM design and memory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that linear recurrent models can approximate optimal HMM belief states in previously studied next-token prediction tasks, and studies when linear models suffice for belief state approximation.

Problem

When are linear sequence models sufficient to represent the optimal belief states (HMM filtering posteriors) learned by next-token prediction, given that complex models like transformers and SSMs are known to represent such belief states?

Approach

Finds that in previously studied tasks, belief-state representations learned during next-token prediction can also be captured by linear recurrent models, and asks under what conditions linear models are sufficient for optimal belief state approximation in the canonical HMM setting.

Surprising findings

• Belief states previously attributed to complex sequence models can also be captured by linear recurrent models in previously studied tasks.

Closest papers by meaning

• Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learning (ICML 2026 · similarity 0.89)
• Self-Supervised Inference in State-Space Models (ICLR 2022 · similarity 0.88)
• Inverse Approximation Theory for Nonlinear Recurrent Neural Networks (ICLR 2024 · similarity 0.87)
• Mechanistic Interpretability of RNNs emulating Hidden Markov Models (NeurIPS 2025 · similarity 0.86)
• DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.86)
• Understanding and Improving Length Generalization in Recurrent Models (ICML 2025 · similarity 0.86)