Research map / Architectures and efficiency / RNNs and normalization
Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations
Behnam Neyshabur, Yuhuai Wu, Ruslan Salakhutdinov, Nati Srebro
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with Path-SGD instead of SGD | single-layer RNN, 200 hidden units · word-level PTB | training and test error | Improved |
| drop second term kappa(2) from Path-SGD update (ablation) | Path-SGD · word-level PTB | training and test error | No clear effect |
| fewer hidden units H and longer length T | Path-SGD second term · PTB | norm ratio of kappa(2) to kappa(1) | Improved |
| Path-SGD with identity-initialized RNN instead of SGD | IRNN · Adding problem T=400 | test MSE: 16.7 -> 0 | Improved |
| Path-SGD with identity-initialized RNN instead of SGD | IRNN · Adding problem T=750 | test MSE: 16.7 -> 0 | Improved |
| increase sequence length to 1000 | RNN-Path · Adding problem | learning success | Hurt |
| Path-SGD instead of SGD for identity-initialized RNN | IRNN · sequential MNIST | test error: 7.1 -> 3.1 | Improved |
| Path-SGD instead of plain training of ReLU RNN | RNN-ReLU · PTB character-level | test BPC: 1.55 -> 1.47 | Improved |
| Path-SGD instead of plain training of ReLU RNN | RNN-ReLU · text8 | test BPC: 1.65 -> 1.58 | Improved |
| Path-SGD ReLU RNN versus tanh RNN | RNN-tanh · text8 | test BPC: 1.70 -> 1.58 | Improved |
| Path-SGD RNN versus TRec beta=500 | RNN-Path · PTB character-level | test BPC: 1.48 -> 1.47 | Improved |
Closest papers by meaning
- Path-conditioned training: a principled way to rescale ReLU neural networks (ICML 2026 · similarity 0.89)
- SBO-RNN: Reformulating Recurrent Neural Networks via Stochastic Bilevel Optimization (NeurIPS 2021 · similarity 0.88)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.87)
- On the Convergence Rate of Training Recurrent Neural Networks (NeurIPS 2019 · similarity 0.87)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.86)
- Activation-Descent Regularization for Input Optimization of ReLU Networks (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
