Research map / Architectures and efficiency / RNNs and normalization
The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the feature speed formula relating feature updates after one GD step to the backward-feature angle, and uses it to characterize hyper-parameter scalings (FSC, depth muP) for deep ReLU MLPs and linear ResNets at large width-then-depth.
Problem
Hyper-parameter tuning (initialization scales, learning rates) only indirectly controls feature learning in deep networks; there is no simple tool to predict and control how much features move during training, especially at large depth.
Approach
An exact identity for one infinitesimal GD step expresses the feature velocity norm as the loss-decay contribution divided by cos(theta) times the backward-pass norm, where theta is the backward-feature angle; the angle is linked to the spectrum of the backward-to-feature kernel and layer-to-layer Jacobian conditioning, which lets HP scalings be derived by enforcing dynamical properties (signal pr
Key results
- In ReLU MLPs with iid initialization the backward-feature angle degenerates as cos(theta_l)=Theta(1/sqrt(l)), whereas linear ResNets with branch scale O(1/sqrt(depth)) keep cos(theta_l)=Theta(1).
- The FSC scaling for deep ReLU MLPs is the unique one (among the considered families) satisfying forward, sensitivity, contribution and loss-decay properties, with output scale sqrt(depth)/width, while MF+muP fails loss decay (decays as L^{-1/2}) and NTK fails feature learning.
- Numerical experiments on ReLU MLPs (width 400) and ResNets confirm the predicted sensitivity scalings: NTK sensitivity ~1/sqrt(m), MF+muP ~sqrt(L), FSC ~1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FSC scaling instead of MF+μP scaling | ReLU MLP · ReLU MLP, dense setting | loss decay (LD) | Helped |
| Use FSC scaling instead of NTK scaling | ReLU MLP · ReLU MLP, dense setting | feature learning (FL) | Helped |
| Use MF+μP scaling | ReLU MLP · ReLU MLP, dense setting | loss decay (LD): -dot L = Θ(L^{-1/2}) | Hurt |
| Use NTK scaling | ReLU MLP · ReLU MLP, dense setting | feature learning (FL): ||dot f_{L-1}||_rms = o(1) | Hurt |
| Set ResNet branch scale to O(1/√L) | Linear ResNet · Linear ResNet, large width and depth | cos(θ_v): Θ(1) | Helped |
| Use ReLU MLP with iid initialization as depth increases | ReLU MLP · ReLU MLP, large width and depth | cos(θ_v): Θ(v^{-1/2}) | Hurt |
Surprising findings
- Standard NTK scaling yields vanishing feature learning in deep MLPs, and MF+muP does not satisfy loss decay at large depth.
Limitations
- Analysis is restricted to (stochastic) gradient descent with infinitesimal step size and to cut nodes of the architecture, not applicable inside ResNet blocks or to adaptive optimizers in current form.
- ReLU MLP results are for single inputs and large width-then-depth limits; ReLU ResNet BFA is only proven for linear activations.
- ReLU MLPs are fundamentally flawed models at large depth due to degeneracy of conjugate kernel and NTK with multiple inputs, so the practical relevance is limited.
Experimental setup
Metrics: backward-feature angle sensitivity of features to loss decay feature update RMS norm
Baselines: NTK parameterization MF+muP
Builds on: Maximal update parameterization (muP) Neural tangent kernel Depth muP for ResNets Polyak step-size Dynamical isometry
Closest papers by meaning
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.89)
- How Feature Learning Can Improve Neural Scaling Laws (ICLR 2025 · similarity 0.89)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.88)
- Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.88)
- The Optimization Landscape of SGD Across the Feature Learning Strength (ICLR 2025 · similarity 0.87)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.