Research map / Architectures and efficiency / RNNs and normalization
Revisiting Glorot Initialization for Long-Range Linear Recurrences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Glorot-initialized linear RNNs become unstable over long sequences (t=O(sqrt(n))) because the spectral radius sits slightly above one, and proposes a dimension-aware variance rescaling that pushes the spectral radius below one with high probability.
Problem
Glorot initialization, derived under the infinite-width fixed-depth regime, is assumed stable for linear RNNs, but repeated application of the same recurrent matrix over long sequences can cause hidden-state explosion.
Approach
The spectral radius of Glorot-initialized Gaussian matrices exceeds one by O(1/sqrt(n)) with Gumbel fluctuations, so a dimension-dependent variance rescaling shifts the radius below one with probability about 0.86 to prevent hidden-state explosion in the double-scaling limit.
Key results
- Standard Glorot initialization fails to train on all three LRA tasks (CIFAR-10, IMDB, ListOps) due to exploding hidden states.
- Rescaled initialization achieves 81.54% on sequential CIFAR-10 vs 76.4% for Glorot/2 (dense), and 47.51% on ListOps (dense), best in 4 of 6 settings.
- The rescaling targets P(spectral radius < 1) ≈ 0.86, versus about 0.61 (real) and 0.37 (complex) for standard Glorot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| rescale dense Glorot instead of halving its variance | Linear RNN · Sequential CIFAR-10 | accuracy: 76.4{ $\pm$ 0.004} -> 81.54{ $\pm$ 0.009} | Helped |
| rescale dense Glorot instead of halving its variance | Linear RNN · IMDB | accuracy: 88.74{ $\pm$0.002} -> 87.55 { $\pm$ 0.002} | Hurt |
| rescale dense Glorot instead of halving its variance | Linear RNN · ListOps | accuracy: 46.21 $\pm$ { 0.045} -> \textbf{47.51} { $\pm$ 0.011} | Helped |
| use rescaled Glorot eigenvalues instead of diagonal Glorot/2 | Diagonal linear RNN · Sequential CIFAR-10 | accuracy: 71.6{ $\pm$ 0.006} -> 79.53{ $\pm$ 0.009} | Helped |
| use rescaled Glorot eigenvalues instead of diagonal Glorot/2 | Diagonal linear RNN · IMDB | accuracy: 86.67{ $\pm$ 0.002} -> \textbf{87.31}{ $\pm$ 0.001} | Helped |
| use rescaled Glorot eigenvalues instead of diagonal Glorot/2 | Diagonal linear RNN · ListOps | accuracy: 49.01{ $\pm$ 0.004} -> \textbf{49.49} { $\pm$ 0.005} | Helped |
| use rescaled Glorot eigenvalues instead of unit-circle diagonal initialization | Diagonal linear RNN · Sequential CIFAR-10 | accuracy: \textbf{81.47}{ $\pm$ 0.002} -> 79.53{ $\pm$ 0.009} | Hurt |
| use rescaled Glorot eigenvalues instead of unit-circle diagonal initialization | Diagonal linear RNN · IMDB | accuracy: 86.34{ $\pm$ 0.004} -> \textbf{87.31}{ $\pm$ 0.001} | Helped |
| use rescaled Glorot eigenvalues instead of unit-circle diagonal initialization | Diagonal linear RNN · ListOps | accuracy: 48.93{ $\pm$ 0.004} -> \textbf{49.49} { $\pm$ 0.005} | Helped |
| use standard Glorot instead of rescaled Glorot | Linear RNN · Long-sequence modeling tasks | trainability | Hurt |
Surprising findings
- Standard Glorot initialization, the common baseline for linear RNNs, fails to train on all tasks because its spectral radius exceeds one.
- Spectral radius of Glorot matrices converges to one from above, so eigenvalues exceed the unit circle with substantial probability even though the bulk lies inside it.
Limitations
- Theory gives only a lower bound on hidden-state norm, under i.i.d. Gaussian inputs and complex Gaussian weights.
- Experiments use small linear RNNs (six layers) without state-of-the-art LRU or SSM comparisons.
- Analysis is limited to behavior at random initialization, not training dynamics.
Experimental setup
Datasets: Sequential CIFAR-10 IMDB ListOps
Benchmarks: Sequential CIFAR-10 IMDB ListOps
Metrics: classification accuracy
Baselines: Glorot Glorot/2 Diag. Uniform (circular-law diagonal init) Diag. Glorot/2
Builds on: Glorot initialization Linear Recurrent Units circular law Long Range Arena
Closest papers by meaning
- How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences (NeurIPS 2026 · similarity 0.87)
- Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.85)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.85)
- Understanding and Improving Length Generalization in Recurrent Models (ICML 2025 · similarity 0.85)
- Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models (ICML 2026 · similarity 0.84)
- Unconditional stability of a recurrent neural circuit implementing divisive normalization (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.