Research map / Optimization / Implicit bias and generalization
Scaling Properties of Deep Residual Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use tanh instead of ReLU in fully connected ResNets, with a shared trainable residual scale | Fully connected ResNets · Synthetic dataset and MNIST | weight scaling regime: δ^(L) ∼ L^{-0.7}; β = 0.3 for synthetic and β = 0.2 for MNIST | Mixed |
| Use ReLU with a layer-dependent signed residual scale instead of tanh with a shared scale | Fully connected ResNets · Synthetic dataset and MNIST | weight scaling regime: L^{0.2} for synthetic; L^{0.1} for MNIST; β=0.8 and 0.9, respectively | Mixed |
| Replace tanh fully connected layers with convolutional layers and ReLU | Residual networks · CIFAR-10 | weight scaling regime: α=0.1 and β = 0 | Mixed |
| Remove the noise component from trained tanh weights (ablation) | Fully connected tanh ResNet · Synthetic dataset training loss | training loss | No clear effect |
| Remove the noise component from trained ReLU weights (ablation) | Fully connected ReLU ResNet · Synthetic dataset training loss | training loss: one order of magnitude above | Hurt |
| Increase convolutional ResNet depth | Convolutional ResNet · CIFAR-10 test | test error (%): 6.64 (L=8) to 6.37 (L=121) | No clear effect |
| Vary batch size while training tanh fully connected ResNets | Tanh ResNet · Synthetic dataset | scaling exponents α and β | No clear effect |
Closest papers by meaning
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.90)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.89)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- Generalization of Scaled Deep ResNets in the Mean-Field Regime (ICLR 2024 · similarity 0.88)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.88)
- Implicit regularization of deep residual networks towards neural ODEs (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.