Research map / Architectures and efficiency / RNNs and normalization
Width and depth of ReLU networks
Determines minimum width and depth needed for ReLU networks to approximate functions.
53 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in RNNs and normalization, keyword cluster “layer · activation · functions”
State History Tried Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
16 papers in 2023–24 and 13 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 2.5% to 1.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · The Expressive Power of Neural Networks: A View from the Width (NeurIPS 2017 · 424 citations)
- 2018 · Efficient Neural Network Robustness Certification with General Activation Functions (NeurIPS 2018 · 332 citations)
- 2019 · Deep ReLU Networks Have Surprisingly Few Activation Patterns (NeurIPS 2019 · 105 citations)
- 2020 · The phase diagram of approximation rates for deep neural networks (NeurIPS 2020 · 53 citations)
- 2021 · Minimum Width for Universal Approximation (ICLR 2021 · 44 citations)
- 2022 · Deep ReLU Networks Preserve Expected Length (ICLR 2022 · 7 citations)
- 2023 · Achieve the Minimum Width of Neural Networks for Universal Approximation (ICLR 2023 · 4 citations)
- 2024 · DeepBern-Nets: Taming the Complexity of Certifying Neural Networks Using Bernstein Polynomial Activations and Precise Bound Propagation (AAAI 2024 · 5 citations)
- 2025 · Minimum Width for Deep, Narrow MLP: A Diffeomorphism Approach (NeurIPS 2025 · 1 citation)
- 2026 · Why ReLU? A Bit-Model Dichotomy for Deep Network Training (ICML 2026 · 0 citations)
Earliest papers here
- The Expressive Power of Neural Networks: A View from the Width (NeurIPS 2017 · 424 citations)
- Efficient Neural Network Robustness Certification with General Activation Functions (NeurIPS 2018 · 332 citations)
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks (ICML 2018 · 143 citations)
What has been tried
38 papers here report 191 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 9%, hurt 16%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 29 papers; from OpenAlex and the papers' affiliations).
- London School of Economics and Political Science · 2 papers
- University of Twente · 2 papers
- Freie Universität Berlin · 2 papers
- Beijing Normal University · 2 papers
- University of California, Los Angeles · 2 papers
- Korea University · 2 papers
- Korea Institute for Advanced Study · 2 papers
- Gwangju Institute of Science and Technology · 2 papers
- University of Technology Nuremberg · 2 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Layer-wise analysis of language models (145 papers)
- Symmetry and equivariance in loss landscapes (95 papers)
- Mechanistic circuit interpretability (93 papers)
- Sparse dictionary representations (98 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Dataset condensation and flow estimation (126 papers)
- Differentiable optimization layers (125 papers)
- Scalable approximation and equivariant compute (120 papers)
- Non-autoregressive translation (102 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).