Research map / Optimization / Implicit bias and generalization

Implicit regularization of deep residual networks towards neural ODEs

ICLR 2024 · Optimization › Implicit bias and generalization · approach: Explicit and implicit regularization interplay · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that gradient-flow-trained residual networks with weight-tied Gaussian initialization converge to neural ODEs as depth grows, for finite training time and, under a linear-overparameterization PL condition, in the long-time limit.

Problem

The link between trained residual networks and neural ODEs lacks a rigorous foundation: it is unknown whether trained residual networks still discretize a neural ODE after training, and whether gradient flow converges to a global minimum in the long-time limit for deep residual networks.

Approach

Scale residual branches by 1/L and learning rates by L, initialize as a discretization of a Lipschitz neural ODE (zero outer weights, weight-tied Gaussian inner weights), and show that the trained weights remain a Lipschitz function of depth; a local PL condition holds when width is linear in sample size, giving convergence and interpolation of the limiting ODE.

Key results

• Trained weights converge at rate 1/L to a Lipschitz neural ODE as depth grows; the maximum distance between successive layers scales as 1/L in experiments.
• Under the local PL condition with width m >= c1 n, the loss decays exponentially, l(t) <= exp(-C't/(n sqrt(n) q)) l(0), and the limiting ODE interpolates the training data.
• On CIFAR-10 with GELU and weight-tied initialization, the trained weights stay smooth, with test accuracy 79.9 ± 0.2 versus 85.7 ± 0.1 for i.i.d. initialization.

Measured effects reported in the paper

Surprising findings

• With ReLU, weight-tied initialization yields higher test accuracy (88.1) but non-smooth trained weights, so the neural ODE structure is lost.
• The neural ODE structure costs accuracy relative to i.i.d. initialization on CIFAR-10 with GELU.

Limitations

• Long-time convergence result requires width m linear in n and a small target norm, and the PL condition is proved only for the specific two-layer MLP residual architecture.
• Smoothness of trained weights for CIFAR-10 is judged by visual inspection, and the i.i.d.-initialization case is not covered by the theory.
• Experiments use convolutional and batch-normalized variants that deviate from the theoretical model.

Experimental setup

Datasets: CIFAR-10 synthetic Gaussian data

Benchmarks: CIFAR-10

Metrics: accuracy training loss weight smoothness (visual)

Baselines: i.i.d. initialization ReLU activation

Builds on: neural ODEs residual networks Polyak-Lojasiewicz condition

Closest papers by meaning

• Do Residual Neural Networks discretize Neural Ordinary Differential Equations? (NeurIPS 2022 · similarity 0.93)
• Neural Ordinary Differential Equations (NeurIPS 2018 · similarity 0.91)
• Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks (NeurIPS 2019 · similarity 0.91)
• On the Implicit Bias of Adam (ICML 2024 · similarity 0.87)
• Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.87)
• Improving Neural ODE Training with Temporal Adaptive Batch Normalization (NeurIPS 2024 · similarity 0.87)