Research map / Architectures and efficiency / RNNs and normalization
Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use nnRNN's Schur-form recurrent connectivity with non-normal terms | RNN on PTB character prediction · PTB test, T=300, fixed hidden units | BPC: 1.55 $\pm$ 0.001 -> 1.49 $\pm$ 0.002 | Helped |
| Use nnRNN's Schur-form recurrent connectivity with non-normal terms | RNN on PTB character prediction · PTB test, T=150, fixed parameters | BPC: 1.49 $\pm$ 0.008 -> 1.47 $\pm$ 0.003 | Helped |
| Use nnRNN instead of orthogonal RNN | Copy task · Copy task, T=200 | cross-entropy loss | No clear effect |
| Use nnRNN instead of orthogonal RNN | Permuted sequential MNIST · Permuted sequential MNIST | accuracy | No clear effect |
| Clamp gamma at 1 rather than use regularized, optimized gamma (ablation) | nnRNN · PTB test, T=150 | BPC: 1.47 $\pm$ 0.003 -> 1.46 $\pm$ 0.005* | Mixed |
| Clamp gamma at 1 rather than use regularized, optimized gamma (ablation) | nnRNN · PTB test, T=300 | BPC: 1.49 $\pm$ 0.002 -> 1.49 $\pm$ 0.022** | Mixed |
| Clamp gamma at 0.958 rather than use regularized, optimized gamma (ablation) | nnRNN · PTB test, T=150 | BPC: 1.47 $\pm$ 0.003 -> 1.47 $\pm$ 0.005 | No clear effect |
| Clamp gamma at 0.958 rather than use regularized, optimized gamma (ablation) | nnRNN · PTB test, T=300 | BPC: 1.49 $\pm$ 0.002 -> 1.49 $\pm$ 0.008 | No clear effect |
Closest papers by meaning
- Improved memory in recurrent neural networks with sequential non-normal dynamics (ICLR 2020 · similarity 0.92)
- Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory (AAAI 2020 · similarity 0.90)
- Orthogonal Recurrent Neural Networks with Scaled Cayley Transform (ICML 2018 · similarity 0.89)
- Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections (ICML 2017 · similarity 0.88)
- Complex Unitary Recurrent Neural Networks Using Scaled Cayley Transform (AAAI 2019 · similarity 0.88)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.