Research map / Architectures and efficiency / RNNs and normalization
Brain-inspired recurrent architectures
Studies recurrent network structure, depth analogies, and biologically constrained models such as Dale's law.
87 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “rnn · long · term”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
14 papers in 2023–24 and 30 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 2.2% to 3.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism (NeurIPS 2016 · 220 citations)
- 2017 · Gated Recurrent Convolution Neural Network for OCR (NeurIPS 2017 · 132 citations)
- 2018 · Sequence-to-Point Learning With Neural Networks for Non-Intrusive Load Monitoring (AAAI 2018 · 548 citations)
- 2019 · Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (ICLR 2019 · 77 citations)
- 2020 · The interplay between randomness and structure during learning in RNNs (NeurIPS 2020 · 46 citations)
- 2021 · Recurrent Independent Mechanisms (ICLR 2021 · 71 citations)
- 2022 · Oscillatory Fourier Neural Network: A Compact and Efficient Architecture for Sequential Processing (AAAI 2022 · 4 citations)
- 2023 · Neural Networks and the Chomsky Hierarchy (ICLR 2023 · 44 citations)
- 2024 · Traveling Waves Encode The Recent Past and Enhance Sequence Learning (ICLR 2024 · 10 citations)
- 2025 · Unfolding the Black Box of Recurrent Neural Networks for Path Integration (NeurIPS 2025 · 3 citations)
- 2026 · On Logical Extrapolation for Mazes with Recurrent and Implicit Networks (AAAI 2026 · 2 citations)
Earliest papers here
- RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism (NeurIPS 2016 · 220 citations)
- Gated Recurrent Convolution Neural Network for OCR (NeurIPS 2017 · 132 citations)
- Recurrent Highway Networks (ICML 2017 · 83 citations)
What has been tried
58 papers here report 551 measured changes (a component added, removed or swapped, compared with a baseline): helped 61%, no clear effect 16%, hurt 19%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| replace the linear output map with a 2-layer MLP | psMNIST · test accuracy | 96.7 -> 97.6 | Helped | Traveling Waves Encode The Recent Past and Enhance Sequence Learning (ICLR 2024 · 10 citations) |
| Use only the network output at the decision time point instead of combining outputs up to decision time | RDM task, all coherence levels · RT-distribution fit | Helped | RTify: Aligning Deep Neural Networks with Human Behavioral Decisions (NeurIPS 2024 · 5 citations) | |
| allow learned rather than fixed intrinsic time constants | sMNIST test · accuracy | 90.3% -> 97.7% | Helped | Unconditional stability of a recurrent neural circuit implementing divisive normalization (NeurIPS 2024 · 2 citations) |
| Constrain bilinear transition matrix to block size 1 | Modular addition, m=50, length 500 · normalized accuracy | 1.00 -> 0.02 | Hurt | Revisiting Bi-Linear State Transitions in Recurrent Neural Networks (NeurIPS 2025 · 1 citation) |
| Fix single-neuron timescales to \tau = 1 rather than training \tau | N-parity · training performance | No clear effect | Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · 1 citation) | |
| Add a post-reaction window | Delayed cue-discrimination · learned mechanism | Helped | Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · 0 citations) | |
| Replace the prescribed flow action on each lifted hidden index with a learned convolutional transformation | KTH action recognition · accuracy | Hurt | Flow Equivariant Recurrent Neural Networks (NeurIPS 2025 · 0 citations) | |
| Use low-rank, low-sparsity recurrent connectivity in CfCs | Seaquest, online under distribution shift · normalized reward | Helped | Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control (ICLR 2024 · 0 citations) | |
| Replace the original ReLU-based conditional operator with the optimized version | Parameter-noise robustness · numerical stability | Helped | Universal In-Context Approximation By Prompting Fully Recurrent Models (NeurIPS 2024 · 0 citations) | |
| replace global max pooling with average pooling | DoorKey · accuracy | 100.00 -> 51.86 | Hurt | NeuralSolver: Learning Algorithms For Consistent and Efficient Extrapolation Across General Tasks (NeurIPS 2024 · 0 citations) |
| Train on multiple features/tasks rather than a single task and feature | N-back, task-irrelevant feature decoding · decoding accuracy | Helped | Geometry of naturalistic object representations in recurrent neural network models of working memory (NeurIPS 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 19 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- Mamba · 3 of 19 papers
- LSTM · 2 of 19 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 44 papers; from OpenAlex and the papers' affiliations).
- Harvard University · 3 papers
- Stanford University · 2 papers
- Salk Institute · 2 papers
- Eberhard-Karls-Universität Tübingen · 2 papers
- Ecole Normale Supérieure de Paris · 2 papers
- Edinburgh University · 2 papers
- IBM · 2 papers
- McGill University · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- RNN training dynamics and gradients (102 papers)
- Efficient recurrent architectures (60 papers)
- LSTM design and memory (54 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Layer-wise analysis of language models (145 papers)
- Dataset condensation and flow estimation (126 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).