Research map / Architectures and efficiency / RNNs and normalization
Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add L1 regularizer on centered activations (with sigma smoothing) to normalization (ablation) | BN, LN, DN · CIFAR-10/100 CNN | accuracy: BN 0.7807→BN* 0.8179; DN 0.8058→DN* 0.8122 (CIFAR-10) | Helped |
| add sigma smoothing term to the normalizer (ablation) | LN · PTB LSTM | test perplexity: LN 119.247 → LN-s 102.492 | Helped |
| add sigma smoothing term to the normalizer (ablation) | BN · PTB ReLU RNN | test perplexity: BN 164.977 → BN-s 139.159 | Helped |
| add sigma smoothing term to the normalizer (ablation) | LN · CIFAR-100 CNN | accuracy: LN 0.4249 → LN-s 0.4863 | Helped |
| add sigma smoothing term to the normalizer (BN-s vs BN) in CIFAR-100 CNN (ablation) | BN · CIFAR-100 CNN | accuracy: BN 0.4814 → BN-s 0.5005 (improves here); CIFAR-10 BN 0.7807→BN-s 0.8017 | Hurt |
| add L1 regularizer to BN (ablation) | BN · PTB LSTM | test perplexity: BN 123.245 → BN +L1 123.736 | Hurt |
| add L1 regularizer to BN (ablation) | BN · PTB ReLU RNN | test perplexity: BN 164.977 → BN +L1 166.658 | Hurt |
| add L1 regularizer to LN | LN · PTB Tanh RNN | test perplexity: LN 154.324 → LN +L1 152.100 (improves slightly); ReLU RNN LN 149.128 → LN +L1 147.937 | Hurt |
| sigma smoothing and L1 combined normalization (LN* and DN*) vs standard LN/BN on PTB | LN, DN · PTB LSTM | test perplexity: LN 119.247 → LN* 101.725; DN* 102.238 vs BN 123.245 | Helped |
| divisive normalization with sigma and L1 (DN*) in place of BN on super-resolution | SRCNN · Set14 x3 PSNR | PSNR: BN 22.31 → DN* 29.38 (SRCNN 29.35) | Helped |
| add BN before ReLU in the super-resolution CNN | SRCNN · Set14 x3 PSNR | PSNR: SRCNN 29.35 → BN 22.31 | Hurt |
| add sigma and L1 DN normalization to CNN on CIFAR-100 (DN* vs BN) | DN · CIFAR-100 CNN | accuracy: DN* 0.5066 vs BN 0.4814 (improves) but DN* 0.8122 vs BN* 0.8179 on CIFAR-10 (slightly lower) | Mixed |
Closest papers by meaning
- Online Normalization for Training Neural Networks (NeurIPS 2019 · similarity 0.92)
- PowerNorm: Rethinking Batch Normalization in Transformers (ICML 2020 · similarity 0.91)
- Four Things Everyone Should Know to Improve Batch Normalization (ICLR 2020 · similarity 0.91)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.91)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.91)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.