Research map / Architectures and efficiency / RNNs and normalization
Improved memory in recurrent neural networks with sequential non-normal dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use chain or chain-with-feedback initialization instead of normal initializers | Vanilla RNN · PTB word | test loss (bpc): 6.557 -> 6.510 | Helped |
| Use chain or chain-with-feedback initialization instead of normal initializers | Vanilla RNN · PTB char. | test loss (bpc): 1.312 -> 1.307 | Helped |
| Use chain-with-feedback initialization instead of normal initializers | Vanilla RNN · enwik8 | test loss (bpc): 1.843 -> 1.774 | Helped |
| Use chain initialization instead of orthogonal initialization | 3-layer LSTM · PTB word | test loss (bpc): 5.937 -> 5.935 | No clear effect |
| Use mixed initialization: chain in the update gate and monotonic recurrent profile in other gates | 3-layer LSTM · PTB char. | test loss (bpc): 1.230 -> 1.227 | Helped |
| Use mixed initialization: chain in the update gate and monotonic recurrent profile in other gates | 3-layer LSTM · enwik8 | test loss (bpc): 1.583 -> 1.577 | Helped |
| Use chain initialization instead of orthogonal initialization | 3-layer LSTM · enwik8 | test loss (bpc): 1.583 -> 1.586 | Hurt |
| Increase feedback strength parameter β (ablation) | Feedback chain RNN · psMNIST | validation loss | Hurt |
| Use chain initialization instead of normal initializers | Vanilla RNN · Copy-500 | successful runs: 0 -> 46 | Helped |
| Use chain-with-feedback initialization instead of normal initializers | Vanilla RNN · Copy-500 | successful runs: 0 -> 11 | Helped |
| Use chain initialization instead of orthogonal initialization | Vanilla RNN · Addition-750 | validation loss | Helped |
Closest papers by meaning
- Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics (NeurIPS 2019 · similarity 0.92)
- Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory (AAAI 2020 · similarity 0.90)
- Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (ICML 2018 · similarity 0.90)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.90)
- Orthogonal Recurrent Neural Networks with Scaled Cayley Transform (ICML 2018 · similarity 0.90)
- On the difficulty of learning chaotic dynamics with RNNs (NeurIPS 2022 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.