Research map / Architectures and efficiency / RNNs and normalization

Rethinking State Tracking in Recurrent Models Through Error Control Dynamics

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Brain-inspired recurrent architectures · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Argues that affine recurrent trackers (SSMs, linear attention-style models) cannot correct hidden-state drift along state-separating directions, so they track only finite horizons governed by a distinguishability ratio, while state-dependent RNNs contract such errors and track to 1000 tokens on group state tracking.

Problem

Recurrent architectures are usually judged by expressivity for state tracking, but robust length generalization of symbolic state tracking fails for many affine models even when they are expressive enough; the paper asks why, focusing on error control of hidden-state drift.

Approach

Proves that an affine return map that exactly preserves all symbolic-state centroids must be the identity on the symbolic subspace spanned by centroid differences, so it cannot contract errors there; accumulated drift then gives a finite horizon T_cross set by when the distinguishability ratio R/M crosses the readability threshold.

Key results

• State-dependent models (tanh RNN, State-gated RNN) reach max-passing length 1000 on C2, C6 and S3 at both depths, while most affine models fail or pass only to 60-500.
• Across 113 S3 models with mp >= 60, T_cross correlates with downstream max-passing length (r=+0.87, p<10^-30 on log-log scale).
• At the max-passing length the median q_t is 0.91 (95% CI [0.83, 1.07]), so failure aligns closer to q=1 than to the nearest-centroid bound of 0.5.

Surprising findings

• Token-gated RNN amplifies hidden-state perturbations (rho_step > 1) yet still reaches length 1000 on C2 and S3 at some depths.
• C2 is a weak test: affine models reach the maximum length through neutral sign-flipping oscillation rather than genuine error correction.

Limitations

• Experiments restricted to finite-group state tracking with small groups; the authors state results do not imply affine models fail in-domain.
• Attention-based baselines are excluded because the study is restricted to recurrent state-update models.
• The Jacobian analysis of normalization operators is an intuition rather than a formal lemma.

Experimental setup

Datasets: synthetic group state-tracking sequences (C2, C6, S3, plus C2xC4 and A4 in appendix)

Benchmarks: synthetic group state tracking C2 C6 S3

Metrics: max-passing length (accuracy >= 90%) distinguishability ratio q(t) normalized perturbation error ratio Pearson correlation of T_cross with max-passing length

Baselines: Mamba Mamba-3 AUSSM Simple AUSSM Negative Mamba Linear RNN Token-gated RNN tanh RNN State-gated RNN PD-SSM DeltaNet DeltaProduct

Builds on: Mamba AUSSM DeltaNet GroupSort

Closest papers by meaning

• Revisiting Bi-Linear State Transitions in Recurrent Neural Networks (NeurIPS 2025 · similarity 0.90)
• The Illusion of State in State-Space Models (ICML 2024 · similarity 0.89)
• (How) Do Language Models Track State? (ICML 2025 · similarity 0.87)
• Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues (ICLR 2025 · similarity 0.87)
• Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing (ICLR 2025 · similarity 0.86)
• The Expressive Capacity of State Space Models: A Formal Language Perspective (NeurIPS 2024 · similarity 0.86)