Research map / Architectures and efficiency / RNNs and normalization
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use minimalRNN instead of the more complex gated RNN architectures | minimalRNN · PTB | perplexity: 101.4; 94.4; 89.9 | Mixed |
| Use critical initialization | vanillaRNN and minimalRNN · sequence MNIST | test accuracy | Helped |
| Use orthogonal recurrent weights under critical initialization (ablation) | vanillaRNN · sequence MNIST | test accuracy | Helped |
| Use orthogonal recurrent weights under critical initialization (ablation) | minimalRNN · sequence MNIST | test accuracy | No clear effect |
| Tie recurrent weights across time (ablation) | minimalRNN · forward signal propagation | memorization timescale | No clear effect |
Closest papers by meaning
- Improved memory in recurrent neural networks with sequential non-normal dynamics (ICLR 2020 · similarity 0.90)
- Efficient recurrent architectures through activity sparsity and sparse back-propagation through time (ICLR 2023 · similarity 0.90)
- Efficient Training of Minimal and Maximal Low-Rank Recurrent Neural Networks (NeurIPS 2025 · similarity 0.89)
- AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks (ICLR 2019 · similarity 0.89)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.89)
- Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.