Research map / Learning theory and algorithms / Kernels and regression theory
Generalization of Scaled Deep ResNets in the Mean-Field Regime
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds generalization bounds for infinitely deep and wide scaled ResNets trained in the mean-field regime, via a time-varying Gram matrix lower bound, KL divergence control, and Rademacher complexity, yielding O(1/sqrt(n)) rates.
Problem
Generalization of deep ResNets beyond the lazy/NTK training regime, where the infinite-depth and infinite-width ResNet training dynamics follow a mean-field PDE gradient flow.
Approach
Replaces the time-invariant NTK Gram matrix with a time-variant, distribution-dependent Gram matrix; lower-bounds its minimum eigenvalue globally, bounds KL divergence of parameter distributions from initialization, and converts this into a Rademacher complexity generalization bound.
Key results
- Proves a global lower bound lambda_min(G_2) >= lambda_0/2 on the Gram matrix along training, with lambda_0 >= Lambda(d) at initialization.
- Shows linear convergence of the empirical loss, L(t) <= exp(-beta^2 Lambda t/(2n)) L(0), with KL(tau_t||tau_0) and KL(nu_t||nu_0) bounded by a constant independent of training time.
- Derives a generalization bound of order O(1/sqrt(n)) (plus a sqrt(log(2/delta)/(2n)) term) for the 0-1 loss via Rademacher complexity, matching NTK-regime rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Choose scaling parameters satisfying the stated condition on \bar{\beta} | Scaled deep ResNet (mean-field) · Training data | empirical loss: \widehat{L}(\tau_t,\nu_t)\leq e^{- \frac{\beta^2 \Lambda}{2n} t }\widehat{L}(\tau_0,\nu_0) | Helped |
| Use the trained mean-field ResNet with \beta>\Omega(\sqrt{n}) | Scaled deep ResNet (mean-field) · Generalization (0-1 classification) | 0-1 error: \lesssim 1/\sqrt{n} + 6 \sqrt{\log(2/\delta)/{2n}} | Helped |
Limitations
- Relies on mild but specific assumptions: non-parallel data points, Gaussian initialization, and smoothness of activations (sigmoid/tanh-like).
- Analysis uses only the lower bound on the G_2 Gram matrix; the G_1 contribution is dropped for proof simplicity.
- Stronger data assumptions (e.g., k-sparse parity) needed to show the distribution moves far from initialization; not covered.
Experimental setup
Builds on: neural tangent kernel mean-field analysis of two-layer networks mean-field ResNet ODE framework Rademacher complexity
Closest papers by meaning
- On global convergence of ResNets: From finite to infinite width using linear parameterization (NeurIPS 2022 · similarity 0.91)
- A Mean Field Analysis Of Deep ResNet And Beyond: Towards Provably Optimization Via Overparameterization From Depth (ICML 2020 · similarity 0.91)
- Branch Scaling Manifests as Implicit Architectural Regularization for Improving Generalization in Overparameterized ResNets (ICML 2026 · similarity 0.90)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.89)
- Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification (NeurIPS 2025 · similarity 0.88)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.