Research map / Optimization / Optimizers
On orthogonality and learning recurrent networks with long term dependencies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Allow singular values to deviate from 1 with margin 0.1 rather than enforcing strict orthogonality (ablation) | Factorized RNN · MNIST | accuracy: 77.18 -> 94.10 | Helped |
| Allow singular values to deviate from 1 with margin 0.1 rather than enforcing strict orthogonality (ablation) | Factorized RNN · pMNIST | accuracy: 83.56 -> 91.44 | Helped |
| Use orthogonal rather than Glorot normal initialization (ablation) | Factorized RNN · pMNIST | accuracy: 79.33 -> 91.44 | Helped |
| Use orthogonal rather than identity initialization (ablation) | Factorized RNN · pMNIST | accuracy: 42.72 -> 91.44 | Helped |
| Use orthogonal rather than Glorot normal initialization (ablation) | Factorized RNN · MNIST | accuracy: 66.71 -> 94.10 | Helped |
| Increase the spectral margin from 0 to 1 (ablation) | Factorized RNN · PTBc-300 | accuracy: 54.88 -> 51.12 | Hurt |
| Increase the spectral margin from 0 to 1 (ablation) | Factorized RNN · PTBc-75 | accuracy: 55.31 -> 57.07 | Helped |
| Use orthogonal rather than Glorot normal initialization (ablation) | Factorized RNN · PTBc-300 | accuracy: 51.04 -> 51.30 | No clear effect |
| Increase the spectral margin to speed optimization (ablation) | Factorized RNN · Copy task, T=200, T=500, T=1000 | convergence speed | Helped |
| Use margin larger than 0.001 instead of margin 0.001 (ablation) | Factorized RNN · Copy task, T=10000 | convergence speed | Hurt |
| Use purely orthogonal or unconstrained transitions rather than intermediate spectral margins (ablation) | Factorized RNN · Adding task, T=1000 | MSE convergence: 2000 epochs | Hurt |
| Use transition nonlinearities such as tanh or ReLU rather than an identity-like transition (ablation) | Factorized RNN · Copy task, T=1000 | task solved | Hurt |
Closest papers by meaning
- Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections (ICML 2017 · similarity 0.90)
- Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group (ICML 2019 · similarity 0.88)
- Coordinate Descent on the Orthogonal Group for Recurrent Neural Network Training (AAAI 2022 · similarity 0.88)
- Orthogonal Weight Normalization: Solution to Optimization Over Multiple Dependent Stiefel Manifolds in Deep Neural Networks (AAAI 2018 · similarity 0.88)
- Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization (NeurIPS 2026 · similarity 0.86)
- Generalization of Scaled Deep ResNets in the Mean-Field Regime (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.