Research map / Architectures and efficiency / RNNs and normalization
Minimum Width for Universal Approximation using Squashable Activation Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that networks with any squashable activation function achieve exact minimum width max{d_x, d_y, 2} (unless d_x=d_y=1) for universal approximation of L^p functions, and gives sufficient conditions (analytic non-affine, piecewise-differentiable with distinct one-sided derivative limits) for squashability.
Problem
Exact minimum width for universal approximation of L^p functions by unbounded-depth networks is known only for ReLU and its variants; general activation functions lack exact characterization.
Approach
Introduces squashable activation functions (alternating compositions with affine maps can approximate the identity and binary step on compact sets) and uses a filling-curve/encoder-decoder construction with width max{d_x,d_y,2} to prove the exact minimum width for all such activations, extending the characterization beyond ReLU variants.
Key results
- For any squashable activation, the minimum width for universal approximation of L^p([0,1]^{d_x}, R^{d_y}) is max{d_x, d_y, 2} unless d_x=d_y=1.
- For d_x=d_y=1 with a monotone squashable activation, the minimum width is 2.
- All non-affine analytic functions and a class of piecewise continuously differentiable functions (e.g., leaky-ReLU, HardSwish) are squashable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a squashable activation function | Fully connected network · L^p([0,1]^{d_x}, R^{d_y}), where d_x\ge2 or d_y\ge2 | minimum width: w_{\sigma}=\max\{d_x,d_y\} | Helped |
| Use a monotone squashable activation function | Fully connected network · L^p([0,1], R) | minimum width: w_{\sigma}=2 | Helped |
| Use a squashable activation function without requiring monotonicity | Fully connected network · L^p([0,1], R) | minimum width: w_{\sigma}\in\{1,2\} | Mixed |
| Extend the activation-function class from ReLU variants to squashable functions, including non-affine analytic functions | Universal approximation networks · L^p([0,1]^{d_x}, R^{d_y}) | minimum width: w_{\min} = \max\{d_x,d_y,2\} | Helped |
Limitations
- Results concern existence of approximators via theoretical constructions, not practical training.
- Non-monotone activations need d_x+d_y>=3 for the stated bound.
- Width-1 case for d_x=d_y=1 is only pinned to {1,2} for non-monotone squashable activations.
Experimental setup
Baselines: ReLU minimum width results Kim et al. leaky-ReLU result Park et al. general activation bounds
Builds on: Park et al. minimum width ReLU result Lu et al. width bounds Kim et al. minimum width for leaky-ReLU
Closest papers by meaning
- Minimum Width for Universal Approximation (ICLR 2021 · similarity 0.93)
- Minimum width for universal approximation using ReLU networks on compact domain (ICLR 2024 · similarity 0.89)
- ReLU Network with Width $d+\mathcal{O}(1)$ Can Achieve Optimal Approximation Rate (ICML 2024 · similarity 0.88)
- Characterizing ResNet's Universal Approximation Capability (ICML 2024 · similarity 0.86)
- Minimum Width for Deep, Narrow MLP: A Diffeomorphism Approach (NeurIPS 2025 · similarity 0.85)
- Approximation theory for 1-Lipschitz ResNets (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.