Research map / Architectures and efficiency / RNNs and normalization
Identity Matters in Deep Learning
Moritz Hardt, Tengyu Ma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the all-convolutional residual architecture with ReLU and no batch normalization, dropout, or max pooling instead of All-CNN | Ours · CIFAR10 | top-1 classification error: 7.25 -> 6.38 | Improved |
| Use the all-convolutional residual architecture with ReLU and no batch normalization, dropout, or max pooling instead of All-CNN | Ours · CIFAR100 | top-1 classification error: 32.39 -> 24.64 | Improved |
| Use the all-convolutional residual architecture with ReLU and no batch normalization, dropout, or max pooling instead of All-CNN | Ours · ImageNet ILSVRC 2012 | top-1 classification error: 41.2 -> 35.29 | Improved |
| Use the all-convolutional residual architecture instead of the original 34-layer counterpart | Ours · ImageNet ILSVRC 2012 | top-1 classification error: 25.03% -> 35.29% | Hurt |
| Use a fixed random projection for the last layer instead of training that layer | Ours · CIFAR10 and CIFAR100 | test error | Improved |
| Initialize residual convolution weights with the smaller standard deviation 1/k^2c | Ours · CIFAR10 and CIFAR100 | training | Improved |
| Use batch normalization in a model of this size | CIFAR10 · CIFAR10 | overfitting | Hurt |
Closest papers by meaning
- Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks (NeurIPS 2020 · similarity 0.89)
- Identity Crisis: Memorization and Generalization Under Extreme Overparameterization (ICLR 2020 · similarity 0.87)
- Batch normalization is sufficient for universal function approximation in CNNs (ICLR 2024 · similarity 0.87)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.86)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.86)
- Most ReLU Networks Admit Identifiable Parameters (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
