Research map / Architectures and efficiency / RNNs and normalization
Recurrent neural networks: vanishing and exploding gradients are not the end of the story
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that RNNs suffer from a "curse of memory": as memory lengthens, hidden states and gradients become increasingly sensitive to parameter changes even without exploding gradients, and that eigenvalue reparametrization, input normalization, and diagonal recurrence (as in LRU/SSMs/GRU) mitigate it.
Problem
Gradient-based learning of recurrent networks is hard for long memories; solving vanishing/exploding gradients may not suffice because parameter sensitivity of hidden states grows as memory lengthens even when dynamics are stable.
Approach
Identifies the 'curse of memory': as the recurrent eigenvalue approaches the unit circle, the sensitivity of hidden states and their gradients to recurrent parameters diverges, even with stable dynamics. Shows that input normalization and eigenvalue reparametrization (plus diagonal recurrence) keep these quantities bounded, explaining why deep SSMs and gated RNNs train better.
Key results
- Analytically, E[(dh_t/dlambda)^2] diverges as |lambda| approaches 1, faster than E[h_t^2], even with stable dynamics.
- In the teacher-student task with nu0=0.99, LRUs significantly outperform fully connected linear RNNs despite having 10 times fewer parameters.
- In deep networks at initialization, LRUs and GRUs keep forward activations and most gradients roughly constant across nu0, while complex diagonal RNNs show exploding gradients.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an LRU instead of a fully connected linear RNN | Linear recurrent teacher-student task · Teacher-student task with long memories | teacher-behavior replication | Helped |
| Make the recurrent weight matrix block diagonal (ablation) | Linear RNN teacher-student task · Teacher-student task, ν₀=0.99 | teacher-behavior replication | Helped |
| Convert 2×2 recurrent blocks and their input and output weights to complex numbers (ablation) | Block-diagonal linear RNN · Teacher-student task, ν₀=0.99 | teacher-behavior replication | No clear effect |
| Add input normalization and exponential eigenvalue parametrization (ablation) | Complex block-diagonal linear RNN · Teacher-student task, ν₀=0.99 | teacher-behavior replication | No clear effect |
| Use a complex diagonal recurrent parametrization instead of fully connected connectivity | Linear RNN trained with Adam · Teacher-student task, ν₀=0.99 | learning speed | Helped |
| Use Adam rather than non-adaptive stochastic gradient descent | Complex diagonal linear RNN · Teacher-student task | learning stability | Helped |
| Use a more concentrated eigenvalue distribution | Complex-valued diagonal network · Teacher-student task | teacher-behavior replication | Hurt |
| Increase the number of hidden neurons | Complex diagonal recurrent network · Teacher-student task | effectiveness of adaptive learning rates | Hurt |
| Use LRU input normalization and eigenvalue parametrization | Complex diagonal recurrent layer · Deep recurrent network initialization | gradient magnitude across characteristic time scales | Helped |
| Use a GRU with Chrono initialization | Deep recurrent network · Next-token prediction initialization | gradient magnitude across characteristic time scales | Helped |
| Add layer normalization (ablation) | Complex diagonal RNN · Deep recurrent network initialization | gradient magnitude | Helped |
| Replace LRU input normalization and eigenvalue parametrization with layer normalization (ablation) | Complex diagonal RNN · Deep recurrent network initialization | learning signal for the smallest eigenvalues | Hurt |
Surprising findings
- Making the recurrence block-diagonal (4x4 blocks) rather than adding the LRU's normalization accounts for most of the performance gap with LRUs.
- Layer normalization mitigates macroscopic gradient explosion but likely kills learning signal for the smallest eigenvalues.
Limitations
- Theory restricted to linear diagonal recurrences with infinite horizon and wide-sense stationary inputs.
- Does not address generalization or memory capacity.
- Analysis only approximates gated architectures such as GRUs and Mamba with input-dependent gates.
Experimental setup
Datasets: synthetic teacher-student linear RNN data natural language sentences for next-token prediction
Benchmarks: synthetic teacher-student task next-token prediction on natural language
Metrics: second moment of hidden states gradient second moment loss Hessian eigenspectrum effective learning rate
Baselines: linear RNN (fully connected) LRU S4 GRU LSTM
Builds on: LRU S4 Chrono initialization Mamba
Closest papers by meaning
- On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis (ICLR 2021 · similarity 0.91)
- AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks (ICLR 2019 · similarity 0.90)
- On the difficulty of learning chaotic dynamics with RNNs (NeurIPS 2022 · similarity 0.90)
- UnICORNN: A recurrent model for learning very long time dependencies (ICML 2021 · similarity 0.90)
- RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients? (ICLR 2020 · similarity 0.90)
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.