Research map / Architectures and efficiency / RNNs and normalization
Recurrent Batch Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply batch normalization to input-to-hidden and hidden-to-hidden LSTM transformations | LSTM · MNIST | accuracy: 98.9 -> 99.0 | Helped |
| Apply batch normalization to input-to-hidden and hidden-to-hidden LSTM transformations | LSTM · pMNIST | accuracy: 90.2 -> 95.4 | Helped |
| Apply recurrent batch normalization | LSTM · Penn Treebank test | bits per character: 1.38 -> 1.32 | Helped |
| Apply recurrent batch normalization | LSTM · text8 test | bits per character: 1.43 -> 1.36 | Helped |
| Add attention batch normalization (ablation) | BN-LSTM Attentive Reader · CNN QA validation | validation curve minimum: 49.5% -> 50.0% | Mixed |
| Add attention batch normalization (ablation) | BN-LSTM Attentive Reader · CNN QA training | convergence speed | Helped |
| Use sequencewise input normalization to handle padding in variable-length sequences (ablation) | BN-everywhere Attentive Reader · CNN QA validation | validation curve minimum: 50.0% -> 47.1% | Helped |
| Make the sequencewise-normalized model bidirectional, reversing only unpadded portions (ablation) | BN-e* Attentive Reader · CNN QA validation | validation curve minimum: 47.1% -> 43.9% | Helped |
| Initialize batch-normalization scale gamma at a small value rather than a higher value (ablation) | BN-LSTM · pMNIST | optimization | Helped |
| Initialize batch-normalization scale gamma at a small value rather than a higher value (ablation) | BN-LSTM · Penn Treebank | optimization | No clear effect |
| Share normalization statistics across timesteps instead of estimating them separately (ablation) | BN-LSTM · recurrent sequence tasks | performance | Hurt |
Closest papers by meaning
- Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (ICLR 2017 · similarity 0.90)
- PowerNorm: Rethinking Batch Normalization in Transformers (ICML 2020 · similarity 0.89)
- Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning (NeurIPS 2017 · similarity 0.88)
- Learning Statistical Scripts with LSTM Recurrent Neural Networks (AAAI 2016 · similarity 0.87)
- MetaNorm: Learning to Normalize Few-Shot Batches Across Domains (ICLR 2021 · similarity 0.87)
- On the State of the Art of Evaluation in Neural Language Models (ICLR 2018 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.