Research map / Architectures and efficiency / RNNs and normalization
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a muP-parameterized residual network with branch scale 1/sqrt(depth) so optimal learning rates transfer across both width and depth, and uses DMFT to show a joint infinite width-and-depth feature-learning limit exists.
Problem
Hyperparameters tuned on small networks transfer across width under muP but not across depth, making tuning of very deep large networks costly.
Approach
Combine muP (mean field) scaling with residual branches scaled by 1/sqrt(L) (and an output scale gamma_0 sqrt(N)) so that feature and prediction updates are independent of both width and depth; characterize the resulting joint limit with dynamical mean field theory.
Key results
- Under the 1/sqrt(L)-muP residual parameterization the optimal learning rate transfers across both depth and width for residual CNNs on CIFAR-10, whereas muP and SP do not transfer across depth.
- Finite-size networks converge to a large width-and-depth proxy: error decays like O(L^-1) at large width, and the initial NTK converges at rate O(L^-2).
- Transfer also holds for ViTs trained with Adam under muP, with and without LayerNorm, and for momentum, weight decay, and warmup/cosine schedules.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use μP with 1/√L-scaled residual branches instead of μP alone | Residual convolutional networks · CIFAR-10 | learning-rate transfer across width and depth | Helped |
| Use μP with 1/√L-scaled residual branches instead of standard parameterization | Residual networks · CIFAR-10 | learning-rate transfer across width and depth | Helped |
| Add BatchNorm to the residual architecture | Standard ResNet · CIFAR-10 | learning-rate transfer consistency across depths | Helped |
| Use 1/√L residual scaling rather than relying on normalization layers | Residual convolutional networks · CIFAR-10 | learning-rate transfer consistency across width and depth | Helped |
| Use 1/√L-scaled residual branches | Residual convolutional networks · Tiny ImageNet | learning-rate transfer across width and depth | Helped |
| Use 1/√L-scaled residual branches | Residual convolutional networks · ImageNet | learning-rate transfer across width and depth | Helped |
| Use 1/√L-scaled residual branches | Vision Transformer with μP and Adam · Tiny ImageNet | learning-rate transfer across width and depth | Helped |
| Use 1/√L-scaled residual branches | Vision Transformer without LayerNorm · ImageNet | learning-rate transfer across depth | Helped |
| Use a linear warm-up learning-rate schedule | Vision Transformer · Tiny ImageNet | learning-rate transfer across settings | Helped |
| Add cosine decay after warm-up | Vision Transformer · Tiny ImageNet | learning-rate transfer across settings | Helped |
| Vary momentum across depths using the proposed parameterization | Residual convolutional network · CIFAR-10 | training-dynamics consistency across depths | Helped |
| Vary feature learning rate γ₀ across depths using the proposed parameterization | Residual convolutional network · CIFAR-10 | training-dynamics consistency across depths | Helped |
Surprising findings
- Adding normalization layers alone only partly fixes depth transfer under muP; the 1/sqrt(L) branch scaling gives more consistent transfer.
Limitations
- Theory is at physics-level rigor, not a formal proof.
- Dataset size and number of optimization steps are treated as O(1); experiments limited to 10-20 epochs.
- Why hyperparameters transfer even when losses differ across width and depth remains theoretically open.
Experimental setup
Datasets: CIFAR-10 Tiny ImageNet ImageNet
Benchmarks: CIFAR-10 Tiny ImageNet ImageNet
Metrics: training loss optimal learning rate transfer across width and depth test/train loss convergence
Baselines: muP standard parameterization (PyTorch default) muP with BatchNorm muP with LayerNorm
Builds on: muP (Tensor Programs V) DMFT for neural network dynamics 1/sqrt(L) residual scaling
Closest papers by meaning
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.91)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.91)
- On the Provable Separation of Scales in Maximal Update Parameterization (ICML 2025 · similarity 0.89)
- Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.89)
- Principled Architecture-aware Scaling of Hyperparameters (ICLR 2024 · similarity 0.88)
- Super Consistency of Neural Network Landscapes and Learning Rate Transfer (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.