AI Paper Atlas

Research map / Learning theory and algorithms / Kernels and regression theory

Are All Layers Created Equal?

Chiyuan Zhang, Samy Bengio, Yoram Singer

JMLR 2022 · Learning theory and algorithms › Kernels and regression theory · approach: Generalization theory of deep networks

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Freeze (do not train) a jointly selected set of residual blocks at random initialization (ablation)ResNet-50 on ImageNet · ImageNet test errorerror rate (%): 38.54 -> 44.36Hurt
Freeze a jointly selected set of residual blocks at random initialization (ablation)ResNet-50 on CIFAR-10 · CIFAR-10 test errorerror rate (%): 8.40 -> 11.74Hurt
Remove a jointly selected set of residual blocks from the network (ablation)ResNet-152 on CIFAR-10 · CIFAR-10 test errorerror rate (%): 8.54 -> 9.23No clear effect
Remove a jointly selected set of residual blocks from the network (ablation)ResNet-50 on ImageNet · ImageNet test errorerror rate (%): 34.74 -> 41.50Hurt
Freeze blocks at random init instead of removing them (keep random projections) (ablation)ResNet-50 on ImageNet · ImageNet test errorerror rate (%): 41.50 -> 44.36Mixed
Add output randomization with random pre-initialized residual blocks (r=4,s=1)ResN(4^2) · CIFAR-10 FGSM attackaccuracy (%): 12.75 -> 69.85Improved
Add output randomization (r=4,s=1)ResN(4^2) · CIFAR-10 PGD attackaccuracy (%): 0.33 -> 6.71Improved
Add output randomization (r=4,s=1)ResN(4^2) · CIFAR-10 clean testaccuracy (%): 91.05 -> 89.45Hurt
Increase randomized stages s from 1 to 4 (ablation)ResN(4^4) with r=4 · CIFAR-10 FGSM attackaccuracy (%): 62.76 -> 69.09Improved
Increase randomized stages s from 1 to 4 (ablation)ResN(4^4) with r=4 · CIFAR-10 clean testaccuracy (%): 89.64 -> 88.24Hurt
Add weight decay and batch normalization to training (ablation)ResNet-50 on ImageNet · ImageNet test errorerror rate (%): 34.8 -> 25.0Improved
Add weight decay and batch normalization to training (ablation)ResNet-152 on CIFAR-10 · CIFAR-10 test errorerror rate (%): 8.5 -> 4.7Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy