Research map / Architectures and efficiency / RNNs and normalization

Learning dynamics in linear recurrent neural networks

ICML 2025 · Architectures and efficiency › RNNs and normalization · approach: Efficient recurrent architectures

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analytically derives learning dynamics of linear RNNs under temporally structured tasks, giving an energy function with an effective regularizer, task-dependent stability/extrapolation results, and an NTK showing recurrence promotes feature learning.

Problem

Learning dynamics of recurrent neural networks on temporally structured data are poorly understood; prior analytical work on linear networks ignores task dynamics and recurrence.

Approach

Decomposes input-output correlation singular values across trajectory time into a constant scale and a temporal component, derives a closed-form energy function for decoupled connectivity modes of aligned linear RNNs, and uses it to explain learning order, stability, extrapolation, and rich feature learning.

Key results

• Learning speed of data singular values is ordered by scale and temporal precedence: larger, later singular values are learned faster in LRNNs, unlike feedforward nets.
• Early-importance task dynamics (decreasing singular values) yield unstable solutions for inverse-exponential (lambda>1) and exponential (lambda<1) dynamics; exponential-dynamics solutions fail to extrapolate to other trajectory lengths.
• Sum-integration networks extrapolate perfectly to other trajectory lengths while mean-integration networks accumulate error, matching theory.

Measured effects reported in the paper

Surprising findings

• Recurrent networks learn later, larger singular values faster, which runs opposite to the largest-first ordering of feedforward linear networks.
• The NTK moves substantially in recurrent networks even at larger initialization scales, indicating recurrence strongly promotes feature learning.

Limitations

• Analysis restricted to linear RNNs with data-aligned weights and constant singular vectors (SVD assumption), with single-output case in main text.
• Results are for linear networks and may not generalize to nonlinear RNNs.
• Local Taylor approximation of recurrent mode dynamics is used for non-exact solutions.

Experimental setup

Datasets: synthetic teacher-generated linear RNN tasks sensory integration task (mean and sum of noisy inputs)

Benchmarks: sensory integration task (mean and sum)

Metrics: training loss singular value convergence to theory extrapolation error across trajectory lengths kernel distance of NTK

Baselines: Schuessler et al. 2020b Smékal et al. 2024

Builds on: Saxe et al. deep linear network learning dynamics Schuessler et al. 2020 low-rank learning in RNNs Neural tangent kernel Atanasov et al. alignment

Closest papers by meaning

• Learning Dynamics of RNNs in Closed-Loop Environments (NeurIPS 2025 · similarity 0.91)
• Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks (NeurIPS 2025 · similarity 0.91)
• Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.90)
• Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.89)
• RNNs perform task computations by dynamically warping neural representations (NeurIPS 2025 · similarity 0.89)
• Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.89)