Research map / Architectures and efficiency / RNNs and normalization

Feature Learning Dynamics in Infinite-Depth Neural Networks

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Representation geometry and collapse · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that reused-weight forward-backward coupling in one-layer ResNets under depth-muP vanishes at initialization as O(1/n) but survives training at infinite width, is suppressed by depth, and motivates neural feature dynamics (NFD), a decoupled forward-backward SDE system with O(L^-1) convergence.

Problem

Infinite-depth feature-learning dynamics of ResNets trained by SGD are not rigorously characterized because backpropagation reuses forward weights via their transpose, creating forward-backward correlations whose role is unclear.

Approach

Uses conditional Gaussian representations to separate reused-weight coupling terms from decoupled Gaussian fluctuations; shows the training-induced coupling is higher order in depth (per-layer O(L^-2)), so a decoupled forward-backward SDE (NFD) is the infinite-depth limit.

Key results

• At initialization, the reused-weight coupling is O(1/n) in coordinate-wise mean square, uniformly over depth, and the forward-backward SDE limit has error O(n^-1 + L^-1).
• During training, the coupling term survives infinite width but its per-layer contribution is O(L^-2), giving accumulated second moment O(L^-2) for NFD convergence.
• Finite-network training dynamics converge to NFD with O(L^-1) depth-discretization error (Corollary, conditional on nondegeneracy assumption).

Surprising findings

• Post-activation ResNets with ReLU can have hidden-state expectations diverge with depth even under 1/sqrt(L) residual scaling.

Limitations

• Analysis limited to one-layer ResNets trained with stream SGD (batch size 1); no Transformers, minibatch, or adaptive optimizers.
• Convergence is sequential (n then L), assumes nondegeneracy of covariance kernels, and holds for a fixed training horizon K.
• Existence of the NFD solution is left to future work.

Experimental setup

Datasets: CIFAR-10

Benchmarks: CIFAR-10

Metrics: W2 distance between finite-network and limit laws minimum covariance eigenvalues coupling term second moment

Builds on: depth-muP maximal update parameterization (muP) Tensor Programs dynamical mean-field theory gradient independence assumption

Closest papers by meaning

• Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.91)
• Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.90)
• The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks (NeurIPS 2024 · similarity 0.89)
• Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.89)
• The Optimization Landscape of SGD Across the Feature Learning Strength (ICLR 2025 · similarity 0.89)
• Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.89)