Research map / Architectures and efficiency / RNNs and normalization
Minimum Width for Deep, Narrow MLP: A Diffeomorphism Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that deep narrow MLPs approximate C^2 diffeomorphisms, reduces uniform-norm minimum width for continuous functions to a geometric quantity w(d_x,d_y), and proves an upper bound max(2d_x+1,d_y)+alpha(sigma) and w(2,2)=4.
Problem
Determining the minimum width of deep, narrow MLPs that achieves the universal approximation property for continuous functions under the uniform norm, where known lower and upper bounds differ significantly.
Approach
Reduce minimum-width universal approximation to a purely geometric function w(d_x,d_y) via approximation of C^2-diffeomorphisms by deep narrow Leaky-ReLU MLPs (single-coordinate transformations), then apply the Whitney embedding theorem to bound w and use a winding-number/homology argument for the width-4 lower bound in 2D.
Key results
- Upper bound on minimum width under uniform norm: max(2d_x+1, d_y) + alpha(sigma), with alpha in {0,1,2} depending on activation (0 for Leaky-ReLU, 1 for ReLU, 2 otherwise).
- For Leaky-ReLU, the minimum width equals the geometric quantity w(d_x,d_y).
- w(2,2)=4, giving a lower bound of 4 for approximating continuous functions from R^2 to R^2 under the uniform norm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the diffeomorphism-based framework to approximate continuous functions | Deep, narrow MLP with Leaky-ReLU · C(K, R^d_y), input dimension d_x | minimum width upper bound | Helped |
| Use width 4 for the two-dimensional input and output case | Deep, narrow MLP with ReLU or Leaky-ReLU · C([0,1]^2, R^2) | minimum width | Helped |
Limitations
- The general upper bound is not shown to be tight in all cases; optimality is established only for specific configurations such as w(2,2)=4 and d_y=2d_x.
- The lower bound proof is specific to input and output dimension two.
- The paper does not provide quantitative approximation rates or experiments.
Experimental setup
Baselines: Park et al. 2021 minimum width ReLU Kim et al. 2023 uniform approximation width Cai 2023 width bounds
Builds on: Whitney embedding theorem invertible neural networks universal approximation Leaky-ReLU deep narrow MLP minimum width results
Closest papers by meaning
- Optimal Minimum Width for the Universal Approximation of Continuously Differentiable Functions by Deep Narrow MLPs (NeurIPS 2025 · similarity 0.95)
- Minimum width for universal approximation using ReLU networks on compact domain (ICLR 2024 · similarity 0.89)
- Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations (ICML 2025 · similarity 0.87)
- ReLU Network with Width $d+\mathcal{O}(1)$ Can Achieve Optimal Approximation Rate (ICML 2024 · similarity 0.87)
- An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network (ICML 2024 · similarity 0.86)
- Characterizing ResNet's Universal Approximation Capability (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.