Research map / Learning theory and algorithms / Sample complexity
Characterizing ResNet's Universal Approximation Capability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical study showing constant-width ResNets (bottleneck ResNets) achieve optimal parameter-count approximation rates for Lipschitz, polynomial, smooth, and Lebesgue-integrable functions, improving over ReLU FNNs by a factor d.
Problem
How many tunable weights a ResNet needs to approximate function classes (polynomials, smooth, Lipschitz, Lebesgue-integrable functions) to error ε, and whether ResNet's approximation rate is optimal.
Approach
Constructs ResNets of constant width (4 neurons per activation layer) with a constant number of nonzero weights per residual block, so identity shortcuts carry the d-dimensional state, giving a factor-d reduction over FNNs and matching the ε-order lower bound for Lebesgue-integrable/Lipschitz functions.
Key results
- Constant-width ResNet (width 4) approximates Lipschitz/Lebesgue-integrable functions with O(c(d)(ε/µ)^(-d/2)) tunable weights, a factor-d reduction vs ReLU FNNs and ε-order optimal.
- Polynomial of degree p approximated by ResNet with O(p log(p/ε)) weights per monomial, vs O(d^2 p log(p/ε)) for ReLU FNN of width O(d).
- On synthetic high-dimensional function, RN(d+1,10,d/10) achieves lower test MSE than NN(d+1,d/10) at d=100,200,300 (e.g., 0.0102 vs 0.0139 at d=100).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the fully connected NN with b-ResNet (width 10) | Function approximation model · Eq. (2), d=100 | MSE loss: 0.0139 -> 0.0102 | Helped |
| Replace the fully connected NN with b-ResNet (width 10) | Function approximation model · Eq. (2), d=200 | MSE loss: 0.0216 -> 0.0131 | Helped |
| Replace the fully connected NN with b-ResNet (width 10) | Function approximation model · Eq. (2), d=300 | MSE loss: 0.0472 -> 0.0225 | Helped |
| Increase ResNet width from 10 to 20 (ablation) | b-ResNet · Eq. (2), d=100 | MSE loss: 0.0102 -> 0.0093 | Helped |
| Increase ResNet width from 10 to 20 (ablation) | b-ResNet · Eq. (2), d=200 | MSE loss: 0.0131 -> 0.0127 | Helped |
| Increase ResNet width from 10 to 20 (ablation) | b-ResNet · Eq. (2), d=300 | MSE loss: 0.0225 -> 0.0228 | Hurt |
| Increase ResNet width from 10 to 40 (ablation) | b-ResNet · Eq. (2), d=100 | MSE loss: 0.0102 -> 0.0091 | Helped |
| Increase ResNet width from 10 to 40 (ablation) | b-ResNet · Eq. (2), d=200 | MSE loss: 0.0131 -> 0.0131 | No clear effect |
| Increase ResNet width from 10 to 40 (ablation) | b-ResNet · Eq. (2), d=300 | MSE loss: 0.0225 -> 0.0230 | Hurt |
Surprising findings
- Constant-width ResNets match the optimal ReLU FNN rate even though identity maps are typically assumed to need width d, challenging the expectation that width must scale with input dimension.
Limitations
- Upper bound for uniform norm not shown for bottleneck ResNets; requires wider network for uniform approximation.
- Constant c(d) in the bound is very large in d; experiments only cover a synthetic function.
- Analysis restricted to ResNet-type FNNs, not convolutional ResNets; trade-off between width and depth left open.
Experimental setup
Benchmarks: synthetic high-dimensional function approximation (Eq. 2), d=100/200/300
Metrics: test MSE
Baselines: Lin & Jegelka 2018 Oono & Suzuki 2019 fully connected ReLU network
Builds on: Lin & Jegelka 2018 ResNet universality Yarotsky 2017 ReLU approximation Shen et al. 2022 optimal ReLU FNN approximation Oono & Suzuki 2019
Closest papers by meaning
- ReLU Network with Width $d+\mathcal{O}(1)$ Can Achieve Optimal Approximation Rate (ICML 2024 · similarity 0.93)
- Minimum width for universal approximation using ReLU networks on compact domain (ICLR 2024 · similarity 0.90)
- Approximation theory for 1-Lipschitz ResNets (NeurIPS 2025 · similarity 0.90)
- A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizations (ICML 2025 · similarity 0.88)
- Generalization of Scaled Deep ResNets in the Mean-Field Regime (ICLR 2024 · similarity 0.87)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.