Research map / Architectures and efficiency / RNNs and normalization
Theory of depth and width
Analyzes exploding gradients, deep linear solutions, and expressive power of deep and wide networks.
223 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “layer · activation · functions”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 5 tracked benchmarks.
60 papers in 2023–24 and 61 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 9.2% to 6.5%.
Open questions the data points to
- ImageNet (Top-1 accuracy): best went from 72.9 (up to 2024) to 76.61 in 2025 — Improving Generalization of Deep Neural Networks by Optimum Shifting (AAAI 2025 · 0 citations)
- ImageNet (Top-5 accuracy): best went from 88.9 (up to 2024) to 93.14 in 2025 — Improving Generalization of Deep Neural Networks by Optimum Shifting (AAAI 2025 · 0 citations)
No longer reported by papers here: CIFAR-10 (test accuracy, last in 2024); CIFAR-10 (accuracy, last in 2024); CIFAR-10 (Top-1 accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Sparsifying Neural Network Connections for Face Recognition (CVPR 2016 · 156 citations)
- 2017 · FractalNet: Ultra-Deep Neural Networks without Residuals (ICLR 2017 · 622 citations)
- 2018 · Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations (ICML 2018 · 153 citations)
- 2019 · Training Neural Networks with Local Error Signals (ICML 2019 · 131 citations)
- 2020 · Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (CVPR 2020 · 124 citations)
- 2021 · Improving Accuracy of Binary Neural Networks Using Unbalanced Activation Distribution (CVPR 2021 · 36 citations)
- 2022 · Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks (NeurIPS 2022 · 17 citations)
- 2023 · Regularization of Polynomial Networks for Image Recognition (CVPR 2023 · 7 citations)
- 2024 · Rewrite the Stars (CVPR 2024 · 612 citations)
- 2025 · KAN: Kolmogorov–Arnold Networks (ICLR 2025 · 535 citations)
- 2026 · Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate (ICML 2026 · 0 citations)
Earliest papers here
- Sparsifying Neural Network Connections for Face Recognition (CVPR 2016 · 156 citations)
- Deep Learning with S-Shaped Rectified Linear Activation Units (AAAI 2016 · 155 citations)
- Maximal Sparsity with Deep Networks? (NeurIPS 2016 · 113 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ImageNet · Top-1 accuracy (higher is better)
6 papers, 2019–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 56.6 | Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization (NeurIPS 2019 · 67 citations) |
| 2021 | 72.9 | How Does Topology Influence Gradient Propagation and Model Performance of Deep Networks With DenseNet-Type Skip Connections? (CVPR 2021 · 12 citations) |
| 2025 | 76.61 | Improving Generalization of Deep Neural Networks by Optimum Shifting (AAAI 2025 · 0 citations) |
ImageNet · Top-5 accuracy (higher is better)
5 papers, 2019–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 79.4 | Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization (NeurIPS 2019 · 67 citations) |
| 2021 | 86.4 | Learning Frequency Domain Approximation for Binary Neural Networks (NeurIPS 2021 · 9 citations) |
| 2023 | 88.9 | Understanding weight-magnitude hyperparameters in training binary networks (ICLR 2023 · 0 citations) |
| 2025 | 93.14 | Improving Generalization of Deep Neural Networks by Optimum Shifting (AAAI 2025 · 0 citations) |
CIFAR-10 · test accuracy (higher is better)
4 papers, 2018–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 82.1 | Learning Deep ResNet Blocks Sequentially using Boosting Theory (ICML 2018 · 30 citations) |
| 2021 | 97 | How Does Topology Influence Gradient Propagation and Model Performance of Deep Networks With DenseNet-Type Skip Connections? (CVPR 2021 · 12 citations) |
| 2024 | 97.23 | $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · 0 citations) |
CIFAR-10 · accuracy (higher is better)
4 papers, 2019–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 84 | From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference (ICLR 2019 · 1 citation) |
| 2021 | 92.54 | Learning Frequency Domain Approximation for Binary Neural Networks (NeurIPS 2021 · 9 citations) |
| 2023 | 94.91 | Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference (ICLR 2023 · 3 citations) |
CIFAR-10 · Top-1 accuracy (higher is better)
4 papers, 2020–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 94.72 | Adaptive Convolutional ReLUs (AAAI 2020 · 10 citations) |
What has been tried
178 papers here report 1,662 measured changes (a component added, removed or swapped, compared with a baseline): helped 62%, no clear effect 12%, hurt 20%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| replace summation with star operations | ImageNet-1k · Top-1 accuracy | 75.3 -> 78.4 | Helped | Rewrite the Stars (CVPR 2024 · 612 citations) |
| Prune the KAN | Feynman I.9.18 · test RMSE | 1.48\times 10^{-3} -> 8.62\times 10^{-3} | Hurt | KAN: Kolmogorov–Arnold Networks (ICLR 2025 · 535 citations) |
| Increase depth | Random-weight functions · Fourier complexity | No clear effect | Neural Redshift: Random Networks are not Random Functions (CVPR 2024 · 9 citations) | |
| Use γ<t instead of γ from all other layers | MNIST · error (%) | 2.1 -> 2.2 | Hurt | Layer Collaboration in the Forward-Forward Algorithm (AAAI 2024 · 6 citations) |
| Add MGDL grades sequentially (grade 1 through grade 4) | Cat, Sea, and Building test images · PSNR | Cat: 20.41 -> 22.67 -> 23.71 -> 24.18; Sea: 18.62 -> 21.50 -> 23.42 -> 24.32; Building: 17.29 -> 18.9 | Helped | Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning (NeurIPS 2024 · 6 citations) |
| Use 4 levels instead of 5 | Navier Stokes NS2 · L² error (×10⁻²) | 1.63 -> 2.10 | Hurt | MgNO: Efficient Parameterization of Linear Operators via Multigrid (ICLR 2024 · 5 citations) |
| Use grid extension | Gaussian random field regression · regression loss | Helped | On the expressiveness and spectral bias of KANs (ICLR 2025 · 4 citations) | |
| Introduce artificially imbalanced neuronal rotation | training experiments · performance | Hurt | Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · 2 citations) | |
| Use non-negative weights with ReLU instead of activation switching | Synthetic function approximation · approximation quality | Hurt | Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations (ICML 2025 · 1 citation) | |
| Random permutation of Mixer (RP-Mixer) vs normal Mixer | CIFAR-10, CIFAR-100, STL-10, ImageNet-1k · test error | Mixed | Understanding MLP-Mixer as a wide and sparse MLP (ICML 2024 · 1 citation) | |
| Remove backpropagation through the DDN layer | small-scale datasets · performance | No clear effect | Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame (NeurIPS 2024 · 1 citation) | |
| Set the SOS batch size to 300 rather than 200 | CIFAR-100 · test accuracy | 72.91 ± 0.19 -> 73.35 ± 0.43 | Helped | Improving Generalization of Deep Neural Networks by Optimum Shifting (AAAI 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 41 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 12 of 41 papers
- MNIST · 9 of 41 papers
- CIFAR-100 · 6 of 41 papers
- ImageNet · 5 of 41 papers
- Fashion-MNIST · 4 of 41 papers
- Tiny-ImageNet · 3 of 41 papers
- SVHN · 2 of 41 papers
- Wikitext-103 · 2 of 41 papers
Baselines
- ReLU MLP · 2 of 41 papers
- Adam · 2 of 41 papers
- MLP · 2 of 41 papers
- maximal update parameterization · 2 of 41 papers
- Orthogonal initialization · 2 of 41 papers
- Xavier initialization · 2 of 41 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 121 papers; from OpenAlex and the papers' affiliations).
- EPFL · 7 papers
- Amazon · 5 papers
- Microsoft · 4 papers
- ETHZ - ETH Zurich · 4 papers
- Swiss Federal Institute of Technology Lausanne · 4 papers
- University of Texas at Austin · 3 papers
- Meta · 3 papers
- Google · 3 papers
- Korea Advanced Institute of Science and Technology · 3 papers
- Princeton University · 3 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Representation geometry and collapse (169 papers)
- Layer-wise analysis of language models (145 papers)
- Symmetry and equivariance in loss landscapes (95 papers)
- Mechanistic circuit interpretability (93 papers)
- Sparse dictionary representations (98 papers)
- Width and depth of ReLU networks (53 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Dataset condensation and flow estimation (126 papers)
- Differentiable optimization layers (125 papers)
- Scalable approximation and equivariant compute (120 papers)
- Non-autoregressive translation (102 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).