Research map / Architectures and efficiency / RNNs and normalization
Representation geometry and collapse
Studies feature redundancy, neural collapse, and initialization effects on generalization in deep networks.
169 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “layer · activation · functions”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
53 papers in 2023–24 and 32 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 8.1% to 3.4%.
Open questions the data points to
- Papers report ResNet50 on ImageNet (Top-1 accuracy (%)) anywhere from 67.3 to 76.1: comparisons against it depend on whose number you take.
No longer reported by papers here: CIFAR-10 test (accuracy, last in 2022); CIFAR-10 (accuracy, last in 2023); CIFAR100 (accuracy (%), last in 2023).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Residual Networks Behave Like Ensembles of Relatively Shallow Networks (NeurIPS 2016 · 596 citations)
- 2017 · Learning Important Features Through Propagating Activation Differences (ICML 2017 · 2,386 citations)
- 2018 · Towards better understanding of gradient-based attribution methods for Deep Neural Networks (ICLR 2018 · 372 citations)
- 2019 · Progressive Image Deraining Networks: A Better and Simpler Baseline (CVPR 2019 · 1,038 citations)
- 2020 · Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks (AAAI 2020 · 86 citations)
- 2021 · Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth (ICLR 2021 · 104 citations)
- 2022 · B-Cos Networks: Alignment Is All We Need for Interpretability (CVPR 2022 · 65 citations)
- 2023 · Flow-Based Robust Watermarking with Invertible Noise Layer for Black-Box Distortions (AAAI 2023 · 65 citations)
- 2024 · The GAN is dead; long live the GAN! A Modern GAN Baseline (NeurIPS 2024 · 31 citations)
- 2025 · BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Models (NeurIPS 2025 · 4 citations)
- 2026 · The Implicit Bias of Depth: From Neural Collapse to Softmax Codes (ICML 2026 · 0 citations)
Earliest papers here
- Residual Networks Behave Like Ensembles of Relatively Shallow Networks (NeurIPS 2016 · 596 citations)
- Analyzing Classifiers: Fisher Vectors and Deep Neural Networks (CVPR 2016 · 209 citations)
- Learning the Number of Neurons in Deep Networks (NeurIPS 2016 · 187 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
CIFAR-10 test · accuracy (higher is better)
5 papers, 2016–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2016 | 94.41 | Blockout: Dynamic Model Selection for Hierarchical Deep Networks (CVPR 2016 · 24 citations) |
| 2021 | 95.9 | EVALUATION OF NEURAL ARCHITECTURES TRAINED WITH SQUARE LOSS VS CROSS-ENTROPY IN CLASSIFICATION TASKS (ICLR 2021 · 51 citations) |
CIFAR-10 · accuracy (higher is better)
4 papers, 2019–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 94.1 | Shallow-Deep Networks: Understanding and Mitigating Network Overthinking (ICML 2019 · 98 citations) |
| 2020 | 95.46 | Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization? (ICML 2020 · 5 citations) |
CIFAR100 · accuracy (%) (higher is better)
4 papers, 2019–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 77.3 | Shallow-Deep Networks: Understanding and Mitigating Network Overthinking (ICML 2019 · 98 citations) |
| 2023 | 92.21 | Lookaround Optimizer: $k$ steps around, 1 step average (NeurIPS 2023 · 2 citations) |
What has been tried
148 papers here report 1,440 measured changes (a component added, removed or swapped, compared with a baseline): helped 55%, no clear effect 13%, hurt 26%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add R2 gradient penalty to RpGAN + R1 | StackedMNIST · training convergence | Helped | The GAN is dead; long live the GAN! A Modern GAN Baseline (NeurIPS 2024 · 31 citations) | |
| leave the last stage unzipped | CIFAR-10 (5+5) · joint accuracy | 79.1\conf{1.1} -> 83.8\conf{3.1} | Helped | ZipIt! Merging Models from Different Tasks without Training (ICLR 2024 · 6 citations) |
| Use P=2 shared gait encoders instead of P=12 | CCPG · rank-1 accuracy (CL) | 89.8 -> 89.5 | No clear effect | BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Models (NeurIPS 2025 · 4 citations) |
| Reduce sign-extraction iterations to s=15 and identify easy and hard neurons | Random models, layer 2 · sign-extraction speedup | up to 14.8 times | Helped | Beyond Slow Signs in High-fidelity Model Extraction (NeurIPS 2024 · 3 citations) |
| Include more targeted layers in the bottleneck | ImageNet-1k val · insertion-deletion difference | Helped | Comprehensive Information Bottleneck for Unveiling Universal Attribution to Interpret Vision Transformers (CVPR 2025 · 2 citations) | |
| Increase Glorot weight magnitude from U(-s,s) to U(-6s,6s) for ReLU | 2D coordinate-grid inputs · ReLU affinity score | 0.9(±0.04) -> 0.82(±0.07) | Helped | From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport (CVPR 2025 · 2 citations) |
| Use sparsity B instead of sparsity A | SRHM · sample complexity scaling | No clear effect | How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Model (ICML 2024 · 2 citations) | |
| Increase k for attention-and-MLP codebooks | TokFSM · State Acc | 63.65 -> 76.32; 63.65 -> 91.53; 63.65 -> 95.82 | Helped | Codebook Features: Sparse and Discrete Interpretability for Neural Networks (ICML 2024 · 2 citations) |
| Sample the entropy landscape using cross-entropy test loss instead of smoothed test accuracy | Modular arithmetic tasks · entropy barrier between memorization and generalization | No clear effect | Is Grokking a Computational Glass Relaxation? (NeurIPS 2025 · 1 citation) | |
| Prune neurons starting from the lowest IPR | Modular Addition, corrupted training examples · accuracy | Hurt | To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · 1 citation) | |
| Add SGR to non-greedy local training | ImageNet · top-1 accuracy | 69.44 -> 70.73 | Helped | Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation (ICML 2024 · 1 citation) |
| Vary κ from 0.04 to 0.03 or 0.05 times the output-score difference | ResNet-20 on CIFAR-10 · interaction extraction robustness | No clear effect | Layerwise Change of Knowledge in Neural Networks (ICML 2024 · 1 citation) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| ResNet50 | ImageNet | Top-1 accuracy (%) | 67.3 – 76.1 | 2 papers, e.g. NeurIPS 2016 and NeurIPS 2022 |
What reviewers will expect
Datasets and baselines named in the 22 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 9 of 22 papers
- ImageNet-1k · 7 of 22 papers
- MNIST · 4 of 22 papers
- SVHN · 2 of 22 papers
- CIFAR-100 · 2 of 22 papers
- TinyImageNet · 2 of 22 papers
- WikiText-103 · 2 of 22 papers
Baselines
- CKA · 2 of 22 papers
- AdamW · 2 of 22 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 85 papers; from OpenAlex and the papers' affiliations).
- Google · 8 papers
- Google DeepMind · 5 papers
- University of Cambridge · 4 papers
- Massachusetts Institute of Technology · 4 papers
- Swiss Federal Institute of Technology · 4 papers
- Georgia Institute of Technology · 3 papers
- Harvard University · 3 papers
- EPFL · 3 papers
- ETHZ - ETH Zurich · 3 papers
- Stanford University · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Theory of depth and width (223 papers)
- Layer-wise analysis of language models (145 papers)
- Symmetry and equivariance in loss landscapes (95 papers)
- Mechanistic circuit interpretability (93 papers)
- Sparse dictionary representations (98 papers)
- Width and depth of ReLU networks (53 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Dataset condensation and flow estimation (126 papers)
- Differentiable optimization layers (125 papers)
- Scalable approximation and equivariant compute (120 papers)
- Non-autoregressive translation (102 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).