AI Paper Atlas

Research map / Learning theory and algorithms / Prediction and decision losses

[Re] Numerical influence of ReLU'(0) on backpropagation

Tommaso Martorella, Hector Manuel Ramirez Contreras, Daniel Garcia

NeurIPS 2023 · Learning theory and algorithms › Prediction and decision losses · approach: Active learning and subset selection

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use ReLU'(0)=1 instead of 0 in 32 bits precision (ablation)fully connected network SGD · MNISTparameter divergence between runsHurt
Increase numerical precision from 32 to 64 bits (ablation)fully connected network on MNIST · bifurcation zone frequencyproportion of weights in S: 40% -> 0%Improved
Decrease precision from 32 to 16 bits (ablation)fully connected network on MNIST · bifurcation zone frequencyproportion of weights in S: 40% -> 100%Hurt
Increase number of training samples (ablation)Monte Carlo bifurcation estimate · MNIST fully connected 4x256proportion of weights in SHurt
Increase mini-batch size (ablation)Monte Carlo bifurcation estimate · MNIST fully connectedproportion of weights in SNo clear effect
Increase network width and depth (ablation)Monte Carlo bifurcation estimate · MNIST fully connectedproportion of weights in SHurt
Set ReLU'(0) to nonzero magnitude above 0.1 (ablation)VGG11 SGD without batch norm · CIFAR10test accuracyHurt
Use ReLU'(0)=1 instead of 0 (ablation)ResNet50 SGD · ImageNettest accuracy: 75% -> 63% or 55%Hurt
Add batch normalization (ablation)VGG11 SGD · CIFAR10sensitivity of test accuracy to ReLU'(0)Improved
Add batch normalization (ablation)ResNet18 SGD · CIFAR10sensitivity of test accuracy to ReLU'(0)Mixed
Use Adam instead of SGD (ablation)VGG11 · CIFAR10 and SVHNsensitivity of test accuracy to ReLU'(0)Improved
Use 64 bits instead of 32 bits with ReLU'(0)=1 (ablation)VGG11 · CIFAR10test accuracyImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy