AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Highway and Residual Networks learn Unrolled Iterative Estimation

Klaus Greff, Rupesh K. Srivastava, Jürgen Schmidhuber

ICLR 2017 · Architectures and efficiency › RNNs and normalization · approach: Representation geometry and collapse · 150 citations (OpenAlex)

This paper tried a common change: Add batch / layer normalization

Use batch normalization · ILSVRC-2012 · top-5 error: 10.03 -> 7.53; 9.40 -> 7.17 Improved

What 431 accepted papers report for this change

See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use learned gating on the identity connection instead of ungated residual connections (ablation)Highway layers in LSTM-Char-Large · Penn Treebank testperplexity: 91.32 -> 79.13Improved
Use coupled transform and carry gates instead of a transform gate alone (ablation)Highway layers in LSTM-Char-Large · Penn Treebank testperplexity: 82.94 -> 79.13Improved
Remove the transform gate while retaining the carry gate (ablation)Highway layers in LSTM-Char-Large · Penn Treebank testperplexity: 79.13 -> 79.15No clear effect
Add a carry gate to the transform-gate-only variant (ablation)Highway layers in LSTM-Char-Large · Penn Treebank testperplexity: 82.94 -> 79.15Improved
Use batch normalization (ablation)50-layer Highway and ResNet · ILSVRC-2012top-5 error: 10.03 -> 7.53; 9.40 -> 7.17Improved
Remove batch normalization (ablation)50-layer Highway and ResNet · ILSVRC-2012training errorImproved
Remove batch normalization (ablation)50-layer Highway and ResNet · ILSVRC-2012top-5 error: 7.53 -> 10.03; 7.17 -> 9.40Hurt
Use Highway-Full instead of Highway (ablation)50-layer Highway with batch normalization · ILSVRC-2012top-5 error: 7.53 -> 7.29No clear effect
Use expressive transform gates with receptive fields matching the primary transformationHighway network · ILSVRC-2012top-5 errorImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy