Research map / Architectures and efficiency / RNNs and normalization
RNN training dynamics and gradients
Analyzes learning, generalization, and vanishing gradients in recurrent networks and proposes stable variants.
102 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “rnn · long · term”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Few recent reports None of the tracked benchmarks has a report from 2025–26.
9 papers in 2023–24 and 14 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 1.4% to 1.5%.
Open questions the data points to
- Papers report Antisymmetric on Noisy-CIFAR (accuracy (%)) anywhere from 48.63 to 54.7: comparisons against it depend on whose number you take.
No longer reported by papers here: Penn Treebank test (perplexity, last in 2023); Penn Treebank validation (perplexity, last in 2023).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Character-Aware Neural Language Models (AAAI 2016 · 1,064 citations)
- 2017 · Full Resolution Image Compression With Recurrent Neural Networks (CVPR 2017 · 942 citations)
- 2018 · A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction (AAAI 2018 · 175 citations)
- 2019 · AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks (ICLR 2019 · 119 citations)
- 2020 · Particle Filter Recurrent Neural Networks (AAAI 2020 · 84 citations)
- 2021 · Lipschitz Recurrent Neural Networks (ICLR 2021 · 47 citations)
- 2022 · projUNN: efficient method for training deep networks with unitary matrices (NeurIPS 2022 · 9 citations)
- 2023 · Minimal Width for Universal Property of Deep RNN (ICML 2023 · 13 citations)
- 2024 · Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · 18 citations)
- 2025 · Revisiting Glorot Initialization for Long-Range Linear Recurrences (NeurIPS 2025 · 1 citation)
- 2026 · Why Are Linear RNNs More Parallelizable? (ICML 2026 · 0 citations)
Earliest papers here
- Character-Aware Neural Language Models (AAAI 2016 · 1,064 citations)
- Architectural Complexity Measures of Recurrent Neural Networks (NeurIPS 2016 · 110 citations)
- Persistent RNNs: Stashing Recurrent Weights On-Chip (ICML 2016 · 67 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Penn Treebank test · perplexity (lower is better)
6 papers, 2016–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2016 | 78.9 | Character-Aware Neural Language Models (AAAI 2016 · 1,064 citations) |
| 2018 | 47.6 | Noisin: Unbiased Regularization for Recurrent Neural Networks (ICML 2018 · 20 citations) |
Penn Treebank validation · perplexity (lower is better)
4 papers, 2017–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 82.9 | Quasi-Recurrent Neural Networks (ICLR 2017 · 325 citations) |
| 2018 | 59.4 | Fraternal Dropout (ICLR 2018 · 8 citations) |
What has been tried
90 papers here report 949 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 8%, hurt 20%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Make the recurrent weight matrix block diagonal | Teacher-student task, ν₀=0.99 · teacher-behavior replication | Helped | Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · 18 citations) | |
| Use sparse instead of dense BPTT | GSCD test · accuracy | 92.5 -> 92.5 | No clear effect | Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training (AAAI 2024 · 4 citations) |
| Add output correction <q, dk> | SlimPajama 340M and 1.3B commonsense reasoning and recall tasks · average accuracy | 42.45 -> 43.36; 28.21 -> 30.40; 49.72 -> 51.78; 34.49 -> 36.37 | Helped | Improving Bilinear RNN with Closed-loop Control (NeurIPS 2025 · 1 citation) |
| Train unsupervised probes on all context combinations instead of Alice easy contexts | Bob hard contexts · probe AUROC | 0.71 -> 0.54; 0.71 -> 0.69 | Hurt | Do Transformer Interpretability Methods Transfer to RNNs? (AAAI 2025 · 0 citations) |
| Increase implicit Mamba2 iterations from 4+1 to 24+4 | D-Pile test · perplexity | 7.97 -> 7.70 | Helped | Implicit Language Models are RNNs: Balancing Parallelization and Expressivity (ICML 2025 · 0 citations) |
| Remove positional embeddings | ImageNet-1k, ViT-T scale · top-1 accuracy | 75.51 -> 75.22 | Hurt | pLSTM: parallelizable Linear Source Transition Mark networks (NeurIPS 2025 · 0 citations) |
| use a Kronecker channel mixer instead of Householder or DPLR mixers | A5 state-tracking · sequence length generalization | Helped | Fixed-Point RNNs: Interpolating from Diagonal to Dense (NeurIPS 2025 · 0 citations) | |
| Remove first-order terms (CPBIRNN instead of CPRNN) | Penn Treebank test, fixed parameter count · BPC | Hurt | A Tensor Decomposition Perspective on Second-order RNNs (ICML 2024 · 0 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| Antisymmetric | Noisy-CIFAR | accuracy (%) | 48.63 – 54.7 | 2 papers, e.g. ICLR 2020 and CVPR 2021 |
What reviewers will expect
Datasets and baselines named in the 11 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- FineWeb · 2 of 11 papers
- SlimPajama · 2 of 11 papers
Baselines
- Mamba-2 · 3 of 11 papers
- Mamba · 3 of 11 papers
- Gated DeltaNet · 3 of 11 papers
- Transformer++ · 3 of 11 papers
- Vim · 2 of 11 papers
- GLA · 2 of 11 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 23 papers; from OpenAlex and the papers' affiliations).
- Max Planck Institute for Intelligent Systems · 3 papers
- Google DeepMind · 2 papers
- ETHZ - ETH Zurich · 2 papers
- University of Oxford · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Brain-inspired recurrent architectures (87 papers)
- Efficient recurrent architectures (60 papers)
- LSTM design and memory (54 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Layer-wise analysis of language models (145 papers)
- Dataset condensation and flow estimation (126 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).