Research map / Architectures and efficiency / RNNs and normalization
Symmetry and equivariance in loss landscapes
Studies parameter symmetries and learns equivariance automatically in neural network architectures.
95 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in RNNs and normalization, keyword cluster “layer · activation · functions”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
23 papers in 2023–24 and 40 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.5% to 4.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (NeurIPS 2017 · 218 citations)
- 2018 · Beyond the Pixel-Wise Loss for Topology-Aware Delineation (CVPR 2018 · 266 citations)
- 2020 · The Diversified Ensemble Neural Network (NeurIPS 2020 · 23 citations)
- 2021 · A Geometric Analysis of Neural Collapse with Unconstrained Features (NeurIPS 2021 · 43 citations)
- 2022 · On Embeddings for Numerical Features in Tabular Deep Learning (NeurIPS 2022 · 58 citations)
- 2023 · REPAIR: REnormalizing Permuted Activations for Interpolation Repair (ICLR 2023 · 11 citations)
- 2024 · Position: Categorical Deep Learning is an Algebraic Theory of All Architectures (ICML 2024 · 2 citations)
- 2025 · A Renormalization Group Framework for Scale-Invariant Feature Learning in Deep Neural Networks (Student Abstract) (AAAI 2025 · 1 citation)
- 2026 · Tversky Neural Networks: Psychologically Plausible Deep Learning with Differentiable Tversky Similarity (ICLR 2026 · 1 citation)
Earliest papers here
- SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (NeurIPS 2017 · 218 citations)
- Deep Subspace Clustering Networks (NeurIPS 2017 · 7 citations)
- On Blackbox Backpropagation and Jacobian Sensing (NeurIPS 2017 · 1 citation)
What has been tried
63 papers here report 566 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 11%, hurt 17%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| remove weight decay | Six regression datasets · NRC1-NRC3 | Hurt | The Prevalence of Neural Collapse in Neural Multivariate Regression (NeurIPS 2024 · 2 citations) | |
| Use full symmetry-aware learned matching instead of permutation-only matching | CIFAR-100 · loss barrier | 0.53 ± 0.07 -> 0.00 ± 0.00 | Helped | Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · 1 citation) |
| rescale the diagonal parameter as √(n−1)θ₁ instead of parameterizing diagonal and off-diagonal components directly | S3 teacher-student setup · shape of the convergence-basin boundary | Helped | A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · 1 citation) | |
| Restrict analysis to decoder layers 11-16 | PASCAL VOC · mIoU | 0.75 -> 0.80 | Helped | Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations (CVPR 2024 · 1 citation) |
| Add the pseudo-completion layer | Synthetic data, most nontrivial cases of K and d · clustering accuracy | an average of 50% | Helped | Deep-Union Completion (AAAI 2025 · 0 citations) |
| Use σ₀=0.01 instead of σ₀=0.1 while applying syre to the projection head | CIFAR-100, linear evaluation, last layer · last-layer accuracy | 31.7% -> 32.5% | Helped | Remove Symmetries to Control Model Expressivity and Improve Optimization (ICLR 2025 · 0 citations) |
| Add subtree flip matching to tree permutation matching | Tabular-Benchmark (16 datasets), test · accuracy barrier | 0.811 ± 0.333 -> 0.455 ± 0.105 | Helped | Linear Mode Connectivity in Differentiable Tree Ensembles (ICLR 2025 · 0 citations) |
| score-based (lowest GIH geometry score) data removal instead of random removal | CIFAR-10, 30% removed · test accuracy (%) | +0.8 (62.5 vs 61.7) | Helped | Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture (ICLR 2025 · 0 citations) |
| Reduce the penultimate-feature dimension from 640 to 10 | CIFAR100 · ECE (%) | 4.11 -> 11.04 | Hurt | Tilt and Average : Geometric Adjustment of the Last Layer for Recalibration (ICML 2024 · 0 citations) |
| Add the alignment loss to training | Sine INRs · alignment performance | Hurt | Equivariant Deep Weight Space Alignment (ICML 2024 · 0 citations) | |
| Increase the Monte-Carlo sample number k from 2 to 8 | Media · MJC | 63.1 ± 1.2 -> 66.8 ± 1.3 | Helped | HORSE: Hierarchical Representation for Large-Scale Neural Subset Selection (NeurIPS 2024 · 0 citations) |
| learn the group representation rather than use a fixed permutation-matrix representation | S10 order prediction · accuracy | 87 ± 2 -> 100 ± 0 | Helped | MatrixNet: Learning over symmetry groups using learned group representations (NeurIPS 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 23 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 5 of 23 papers
- CIFAR-100 · 4 of 23 papers
- MNIST · 4 of 23 papers
- Fashion-MNIST · 3 of 23 papers
Baselines
- MLP · 2 of 23 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 63 papers; from OpenAlex and the papers' affiliations).
- Google · 3 papers
- Bar-Ilan University · 3 papers
- NVIDIA · 3 papers
- Massachusetts Institute of Technology · 3 papers
- University of Wisconsin - Madison · 2 papers
- Korea Advanced Institute of Science and Technology · 2 papers
- Peking University · 2 papers
- Technion - Israel Institute of Technology · 2 papers
- Northeastern University · 2 papers
- Imperial College London · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Layer-wise analysis of language models (145 papers)
- Mechanistic circuit interpretability (93 papers)
- Sparse dictionary representations (98 papers)
- Width and depth of ReLU networks (53 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Dataset condensation and flow estimation (126 papers)
- Differentiable optimization layers (125 papers)
- Scalable approximation and equivariant compute (120 papers)
- Non-autoregressive translation (102 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).