Research map / Architectures and efficiency / RNNs and normalization
Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Classifies depthwise parametrizations of residual networks by their infinite-width-then-depth limits, identifies Depth-μP (branch multiplier 1/sqrt(depth)) as optimal for block depth 1, and shows fundamental limitations for block depth ≥2.
Problem
Depth scaling of residual networks: how to choose block multipliers and learning rates as depth grows so that features are stable, nontrivial, feature-diverse, and hyperparameters transfer across depth.
Approach
Extends μP to depth via infinite-width-then-depth limits: classifies all (branch multiplier L^-alpha, learning-rate L^-gamma) scalings, identifies alpha=gamma=1/2 (Depth-μP) as the unique optimal choice for single-layer blocks, proposes absolute-value nonlinearity to maximize feature diversity, and proves that block depth ≥2 cannot achieve the same optimality.
Key results
- Depth-μP (branch multiplier 1/sqrt(depth), learning rate scaling) is shown empirically to admit depthwise hyperparameter transfer for block-depth-1 resnets.
- Absolute-value nonlinearity improves performance of deep block-depth-1 resnets over ReLU and pre-nonlinearity variants on CIFAR-10 with Adam.
- For block depth 2, the optimal learning-rate/multiplier slope shifts from about -2 at depth 2 to about -1 at depth 2^10, indicating no hyperparameter transfer with 1/sqrt(L) scaling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ReLU with absolute value | Depth-μP block-depth-1 ResNet · CIFAR-10 | training performance | Helped |
| place the nonlinearity before the weights (pre-nonlinear ResNet) instead of after them (ablation) | Depth-μP ResNet · CIFAR-10 | minimal log loss | Hurt |
| increase depth beyond 8 layers | Depth-μP pre-nonlinear ResNet · CIFAR-10 | minimal log loss | No clear effect |
| use block depth 2 instead of block depth 1 (ablation) | ReLU ResNet without layer normalization · CIFAR-10 | minimal log loss | Hurt |
| use block depth 2 instead of block depth 1 (ablation) | Absolute-value ResNet with layer normalization · CIFAR-10 | minimal log loss | Hurt |
| use 1/L block scaling (ODE scaling) instead of Depth-μP scaling (ablation) | ResNet · depth scaling experiments | optimal learning rate transfer | Hurt |
| increase the number of blocks under 1/√L scaling (ablation) | Block-depth-2 ResNet · CIFAR-10 | hyperparameter transfer | Hurt |
Surprising findings
- Feature learning alone does not imply hyperparameter transfer: the 1/sqrt(L) scaling yields a feature-learning infinite-depth limit yet optimal hyperparameters still shift for block depth ≥2.
- Block depth 2 networks perform worse than block depth 1 networks at matched parameter count, and training longer does not always close the gap.
Limitations
- Theory for general nonlinear case is given as heuristic claims; rigorous proof only for the linear case.
- Block depth ≥2 results are negative and largely empirical on small MLP and transformer setups.
- Infinite-depth results assume width goes to infinity before depth (1 << depth << width).
Experimental setup
Datasets: CIFAR-10 Common Crawl
Benchmarks: CIFAR-10 Common Crawl
Metrics: log loss validation loss
Baselines: Standard parametrization depth scaling ODE scaling (1/L block multiplier) Pre-nonlinearity ResNet ReLU ResNet
Builds on: μP Tensor Programs ResNet Megatron
Closest papers by meaning
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.89)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.89)
- Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025 · similarity 0.88)
- $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks (NeurIPS 2025 · similarity 0.88)
- The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks (NeurIPS 2024 · similarity 0.88)
- On Feature Learning in Structured State Space Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.