Research map / Architectures and efficiency / RNNs and normalization
Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers
Guodong Zhang, Aleksandar Botev, James Martens
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace EOC with TAT for Leaky ReLU | 50-layer vanilla network · ImageNet, K-FAC | top-1 accuracy: 72.6 -> 74.6 | Improved |
| Replace EOC with TAT for Leaky ReLU | 50-layer vanilla network · ImageNet, SGD | top-1 accuracy: 63.7 -> 71.0 | Improved |
| Replace EOC with TAT for Leaky ReLU | 101-layer vanilla network · ImageNet, K-FAC | top-1 accuracy: 71.8 -> 74.2 | Improved |
| Replace EOC with TAT for Leaky ReLU | 101-layer vanilla network · ImageNet, SGD | top-1 accuracy: 41.6 -> 70.0 | Improved |
| Add rescaled shortcut branches with weight w=0.8 (ablation) | TAT ResNet-50 · ImageNet, K-FAC, 90 epochs | top-1 accuracy: 74.6 -> 76.4 | Improved |
| Add rescaled shortcut branches with weight w=0.8 (ablation) | TAT ResNet-50 · ImageNet, SGD, 90 epochs | top-1 accuracy: 71.0 -> 76.0 | Improved |
| Add rescaled shortcut branches with weight w=0.8 (ablation) | TAT ResNet-101 · ImageNet, K-FAC, 90 epochs | top-1 accuracy: 74.2 -> 77.8 | Improved |
| Add rescaled shortcut branches with weight w=0.8 (ablation) | TAT ResNet-101 · ImageNet, SGD, 90 epochs | top-1 accuracy: 70.0 -> 77.3 | Improved |
| Replace TReLU with PReLU initialized at 0.0 (ablation) | 50-layer vanilla network · ImageNet, K-FAC | top-1 accuracy: 74.6 -> 72.5 | Hurt |
| Replace TReLU with PReLU initialized at 0.25 (ablation) | 50-layer vanilla network · ImageNet, K-FAC | top-1 accuracy: 74.6 -> 73.6 | Hurt |
| Train the TAT-initialized negative slope instead of fixing it (ablation) | TReLU networks · ImageNet | top-1 accuracy | No clear effect |
| Add batch normalization to shortcut-free networks (ablation) | 50- and 101-layer vanilla networks · ImageNet | top-1 accuracy: 72.6 vs 72.8; 71.8 vs 67.6 | No clear effect |
Closest papers by meaning
- Deep Equals Shallow for ReLU Networks in Kernel Regimes (ICLR 2021 · similarity 0.89)
- Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation (ICLR 2023 · similarity 0.87)
- Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts (NeurIPS 2020 · similarity 0.86)
- Understanding Deep Neural Networks with Rectified Linear Units (ICLR 2018 · similarity 0.86)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.85)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
