Research map / Architectures and efficiency / RNNs and normalization
Recurrent Highway Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase recurrence depth from 1 to 10 while keeping the parameter budget fixed (ablation) | RHN · Penn Treebank word-level language modeling | test perplexity: 90.6 to 65.4 | Helped |
| Add weight tying between input and output mappings (ablation) | Variational RHN · Penn Treebank word-level language modeling | test perplexity: 68.5 -> 65.4 | Helped |
| Reduce weight decay for the best 10-layer model (ablation) | RHN · Penn Treebank word-level language modeling | validation/test perplexity: 67.9/65.4 | Helped |
| Increase recurrence depth from 5 to 10 (ablation) | RHN · enwik8 test | BPC: 1.31 -> 1}.{\bf 30 | Helped |
| Increase recurrence depth from 1 to 10 (ablation) | DT-RNN and DT(S)-RNN · JSB Chorales next-step prediction | optimization performance | Hurt |
| Increase recurrence depth (ablation) | RHN · JSB Chorales next-step prediction | optimization performance | No clear effect |
| Push any one Highway layer toward carry behavior, effectively removing its transformation (ablation) | RHN with recurrence depth 6 · JSB Chorales | training performance | Hurt |
| Use a recurrence depth of 10 rather than 5 (ablation) | RHN · text8 test | BPC: 1}.{f 29 vs 1.27 | No clear effect |
Closest papers by meaning
- Language Modeling with Recurrent Highway Hypernetworks (NeurIPS 2017 · similarity 0.91)
- Fast-Slow Recurrent Neural Networks (NeurIPS 2017 · similarity 0.89)
- Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling (AAAI 2018 · similarity 0.88)
- Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (ICML 2018 · similarity 0.88)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.87)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.