Research map / Learning theory and algorithms / Kernels and regression theory
Are All Layers Created Equal?
Chiyuan Zhang, Samy Bengio, Yoram Singer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Freeze (do not train) a jointly selected set of residual blocks at random initialization (ablation) | ResNet-50 on ImageNet · ImageNet test error | error rate (%): 38.54 -> 44.36 | Hurt |
| Freeze a jointly selected set of residual blocks at random initialization (ablation) | ResNet-50 on CIFAR-10 · CIFAR-10 test error | error rate (%): 8.40 -> 11.74 | Hurt |
| Remove a jointly selected set of residual blocks from the network (ablation) | ResNet-152 on CIFAR-10 · CIFAR-10 test error | error rate (%): 8.54 -> 9.23 | No clear effect |
| Remove a jointly selected set of residual blocks from the network (ablation) | ResNet-50 on ImageNet · ImageNet test error | error rate (%): 34.74 -> 41.50 | Hurt |
| Freeze blocks at random init instead of removing them (keep random projections) (ablation) | ResNet-50 on ImageNet · ImageNet test error | error rate (%): 41.50 -> 44.36 | Mixed |
| Add output randomization with random pre-initialized residual blocks (r=4,s=1) | ResN(4^2) · CIFAR-10 FGSM attack | accuracy (%): 12.75 -> 69.85 | Improved |
| Add output randomization (r=4,s=1) | ResN(4^2) · CIFAR-10 PGD attack | accuracy (%): 0.33 -> 6.71 | Improved |
| Add output randomization (r=4,s=1) | ResN(4^2) · CIFAR-10 clean test | accuracy (%): 91.05 -> 89.45 | Hurt |
| Increase randomized stages s from 1 to 4 (ablation) | ResN(4^4) with r=4 · CIFAR-10 FGSM attack | accuracy (%): 62.76 -> 69.09 | Improved |
| Increase randomized stages s from 1 to 4 (ablation) | ResN(4^4) with r=4 · CIFAR-10 clean test | accuracy (%): 89.64 -> 88.24 | Hurt |
| Add weight decay and batch normalization to training (ablation) | ResNet-50 on ImageNet · ImageNet test error | error rate (%): 34.8 -> 25.0 | Improved |
| Add weight decay and batch normalization to training (ablation) | ResNet-152 on CIFAR-10 · CIFAR-10 test error | error rate (%): 8.5 -> 4.7 | Improved |
Closest papers by meaning
- Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization) (NeurIPS 2022 · similarity 0.88)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.87)
- Layer-Aware Analysis of Catastrophic Overfitting: Revealing the Pseudo-Robust Shortcut Dependency (ICML 2024 · similarity 0.87)
- Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks (ICLR 2025 · similarity 0.86)
- Interpreting Robustness Proofs of Deep Neural Networks (ICLR 2024 · similarity 0.86)
- If You Want to Be Robust, Be Wary of Initialization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
