Research map / Architectures and efficiency / RNNs and normalization
Highway and Residual Networks learn Unrolled Iterative Estimation
Klaus Greff, Rupesh K. Srivastava, Jürgen Schmidhuber
This paper tried a common change: Add batch / layer normalization
Use batch normalization · ILSVRC-2012 · top-5 error: 10.03 -> 7.53; 9.40 -> 7.17 Improved
What 431 accepted papers report for this change
- 313 improved
- 39 no clear effect
- 13 mixed
- 66 hurt
- MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers ICML 2026 · use independent layer norm per FFN · training convergence · convergenceImproved
- Angel or Demon: Investigating the Plasticity Interventions' Impact on Backdoor Threats in Deep Reinforcement Learning ICML 2026 · apply Layer Normalization in TM-Scratch · Acrobot and MountainCar · BTP: reduces BTP by over 30%Hurt
- MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning ICML 2026 · add layer normalization · MEAL Level 1 · soup deliveredImproved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned gating on the identity connection instead of ungated residual connections (ablation) | Highway layers in LSTM-Char-Large · Penn Treebank test | perplexity: 91.32 -> 79.13 | Improved |
| Use coupled transform and carry gates instead of a transform gate alone (ablation) | Highway layers in LSTM-Char-Large · Penn Treebank test | perplexity: 82.94 -> 79.13 | Improved |
| Remove the transform gate while retaining the carry gate (ablation) | Highway layers in LSTM-Char-Large · Penn Treebank test | perplexity: 79.13 -> 79.15 | No clear effect |
| Add a carry gate to the transform-gate-only variant (ablation) | Highway layers in LSTM-Char-Large · Penn Treebank test | perplexity: 82.94 -> 79.15 | Improved |
| Use batch normalization (ablation) | 50-layer Highway and ResNet · ILSVRC-2012 | top-5 error: 10.03 -> 7.53; 9.40 -> 7.17 | Improved |
| Remove batch normalization (ablation) | 50-layer Highway and ResNet · ILSVRC-2012 | training error | Improved |
| Remove batch normalization (ablation) | 50-layer Highway and ResNet · ILSVRC-2012 | top-5 error: 7.53 -> 10.03; 7.17 -> 9.40 | Hurt |
| Use Highway-Full instead of Highway (ablation) | 50-layer Highway with batch normalization · ILSVRC-2012 | top-5 error: 7.53 -> 7.29 | No clear effect |
| Use expressive transform gates with receptive fields matching the primary transformation | Highway network · ILSVRC-2012 | top-5 error | Improved |
Closest papers by meaning
- Residual Connections Encourage Iterative Inference (ICLR 2018 · similarity 0.90)
- Accelerated training through iterative gradient propagation along the residual path (ICLR 2025 · similarity 0.87)
- Residual Alignment: Uncovering the Mechanisms of Residual Networks (NeurIPS 2023 · similarity 0.87)
- Multi-level Residual Networks from Dynamical Systems View (ICLR 2018 · similarity 0.87)
- Improving Gradient Flow with Unrolled Highway Expectation Maximization (AAAI 2021 · similarity 0.86)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
