Research map / Architectures and efficiency / RNNs and normalization
Minimum width for universal approximation using ReLU networks on compact domain
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows the exact minimum width for L^p approximation of L^p([0,1]^dx, R^dy) by ReLU-like networks is max{dx,dy,2}, and proves a lower bound d_y+1 for uniform approximation when dx<dy<=2dx, giving a dichotomy between L^p and uniform approximation.
Problem
Determining the exact minimum width enabling universal approximation of width-bounded networks, especially on compact domains and for L^p versus uniform approximation.
Approach
A coding-scheme construction (encoder to scalar codewords, decoder back to vectors) built from ReLU networks of width max{dx,2} for the encoder exploits compactness of the domain to achieve width max{dx,dy,2}; a topological (Brouwer fixed-point) argument gives a width lower bound for uniform approximation by injective-approximable activations.
Key results
- ReLU networks of width max{dx,dy,2} are dense in L^p([0,1]^dx, R^dy), matching the exact minimum width.
- For ReLU-like activations (zeta, Leaky-ReLU, ELU, CELU, SELU, and GELU/SiLU/Mish when dx+dy>=3), the same minimum width max{dx,dy,2} holds.
- For uniform approximation with continuous injection-approximable activations, w_min >= dy+1 when dx<dy<=2dx, tight for Leaky-ReLU when dy=2dx.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a compact domain [0,1]^{d_x} instead of R^{d_x} | ReLU networks for L^p approximation · L^p approximation, when d_x+1>d_y | minimum width w_min: w_{\min}=\max\{d_x+1,d_y\} -> w_{\min}=\max\{d_x,d_y,2\} | Helped |
| Use a ReLU-Like activation instead of ReLU | Networks for L^p approximation on [0,1]^{d_x} · L^p([0,1]^{d_x}, R^{d_y}) | minimum width w_min | No clear effect |
| Use uniform approximation instead of L^p approximation | Networks with continuous injection-approximable activations · C([0,1]^{d_x}, R^{d_y}), d_x < d_y <= 2d_x | minimum width w_min: w_{\min}=\max\{d_x,d_y,2\} -> w_{\min}\ge d_y+1 | Hurt |
Surprising findings
- On a compact domain, ReLU networks need smaller width for L^p approximation than on R^dx when dx+1>dy, showing a dichotomy with the unbounded-domain result.
Limitations
- Lower bound for uniform approximation only covers dx<dy<=2dx; gap remains elsewhere.
- Results are theoretical; no experiments.
Experimental setup
Baselines: Park et al. 2021 ReLU minimum width Kim et al. 2023 Leaky-ReLU minimum width Hanin and Sellke Lu et al. 2017
Builds on: Park et al. minimum width ReLU L^p Lu et al. width bounds Kim et al. Leaky-ReLU width
Closest papers by meaning
- Achieve the Minimum Width of Neural Networks for Universal Approximation (ICLR 2023 · similarity 0.94)
- ReLU Network with Width $d+\mathcal{O}(1)$ Can Achieve Optimal Approximation Rate (ICML 2024 · similarity 0.93)
- Characterizing ResNet's Universal Approximation Capability (ICML 2024 · similarity 0.90)
- Minimum Width for Universal Approximation using Squashable Activation Functions (ICML 2025 · similarity 0.89)
- Minimum Width for Deep, Narrow MLP: A Diffeomorphism Approach (NeurIPS 2025 · similarity 0.89)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.