Research map / Architectures and efficiency / RNNs and normalization
Exploring Normalization in Deep Residual Networks with Concatenated Rectified Linear Units
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ReLU with CReLU in BatchNorm ResNet (ablation) | ResNet-110 · CIFAR-10 | test error: 6.35 -> 5.72 | Helped |
| replace ReLU with CReLU in BatchNorm ResNet (ablation) | ResNet-110 · CIFAR-100 | test error: 27.46 -> 25.89 | Helped |
| replace ReLU with CReLU in BatchNorm ResNet (ablation) | ResNet-50 · ImageNet | top-1 error: 24.01 -> 23.03 | Helped |
| replace ReLU with CReLU at similar parameter count (ablation) | BatchNorm ResNet · CIFAR-10 | test error: 6.35 -> 5.87 | Helped |
| add BatchNorm to plain ReLU ResNet (ablation) | ResNet-110 · CIFAR-100 | test error: 36.10 -> 27.46 | Helped |
| naive NormProp instead of BatchNorm (ablation) | ResNet-110 ReLU · CIFAR-100 | test error: 27.46 -> 37.60 | Hurt |
| remove constant affine terms from NormProp (NormProp2) (ablation) | ResNet-110 ReLU · CIFAR-100 | test error: 37.60 -> 31.71 | Helped |
| replace ReLU with CReLU in NormProp2 ResNet (ablation) | ResNet-110 · CIFAR-100 | test error: 31.71 -> 29.29 | Helped |
| fix gamma (NormProp3) over NormProp2 with CReLU (ablation) | ResNet-34 · ImageNet | top-1 error: 29.31 -> 28.28 | Helped |
| add bias to plain ResNet (ablation) | ReLU vs CReLU · CIFAR-10 | test error: 12.16 -> 7.78 (ReLU); 7.17 -> 6.86 (CReLU) | Mixed |
| double the number of filters in BatchNorm ReLU ResNet (ablation) | ResNet-110 · CIFAR-100 | test error: 27.46 -> 27.33 | No clear effect |
| NormProp variants instead of BatchNorm (ablation) | ResNet-50 CReLU · ImageNet | top-1 error: 23.03 -> 26.12 | Hurt |
Closest papers by meaning
- Deconstructing the Regularization of BatchNorm (ICLR 2021 · similarity 0.90)
- Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning (NeurIPS 2021 · similarity 0.90)
- Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks (NeurIPS 2020 · similarity 0.90)
- Characterizing signal propagation to close the performance gap in unnormalized ResNets (ICLR 2021 · similarity 0.90)
- Kalman Normalization: Normalizing Internal Representations Across Network Layers (NeurIPS 2018 · similarity 0.89)
- Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.