Research map / Learning theory and algorithms / Prediction and decision losses
[Re] Numerical influence of ReLU'(0) on backpropagation
Tommaso Martorella, Hector Manuel Ramirez Contreras, Daniel Garcia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReLU'(0)=1 instead of 0 in 32 bits precision (ablation) | fully connected network SGD · MNIST | parameter divergence between runs | Hurt |
| Increase numerical precision from 32 to 64 bits (ablation) | fully connected network on MNIST · bifurcation zone frequency | proportion of weights in S: 40% -> 0% | Improved |
| Decrease precision from 32 to 16 bits (ablation) | fully connected network on MNIST · bifurcation zone frequency | proportion of weights in S: 40% -> 100% | Hurt |
| Increase number of training samples (ablation) | Monte Carlo bifurcation estimate · MNIST fully connected 4x256 | proportion of weights in S | Hurt |
| Increase mini-batch size (ablation) | Monte Carlo bifurcation estimate · MNIST fully connected | proportion of weights in S | No clear effect |
| Increase network width and depth (ablation) | Monte Carlo bifurcation estimate · MNIST fully connected | proportion of weights in S | Hurt |
| Set ReLU'(0) to nonzero magnitude above 0.1 (ablation) | VGG11 SGD without batch norm · CIFAR10 | test accuracy | Hurt |
| Use ReLU'(0)=1 instead of 0 (ablation) | ResNet50 SGD · ImageNet | test accuracy: 75% -> 63% or 55% | Hurt |
| Add batch normalization (ablation) | VGG11 SGD · CIFAR10 | sensitivity of test accuracy to ReLU'(0) | Improved |
| Add batch normalization (ablation) | ResNet18 SGD · CIFAR10 | sensitivity of test accuracy to ReLU'(0) | Mixed |
| Use Adam instead of SGD (ablation) | VGG11 · CIFAR10 and SVHN | sensitivity of test accuracy to ReLU'(0) | Improved |
| Use 64 bits instead of 32 bits with ReLU'(0)=1 (ablation) | VGG11 · CIFAR10 | test accuracy | Improved |
Closest papers by meaning
- Numerical influence of ReLU’(0) on backpropagation (NeurIPS 2021 · similarity 0.89)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.83)
- Activation-Descent Regularization for Input Optimization of ReLU Networks (ICML 2024 · similarity 0.82)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.82)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.82)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
