Research map / Architectures and efficiency / RNNs and normalization
LSTM design and memory
Improves LSTMs with tensorization, binary gates, sparse structure, and analysis of long memory.
54 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “rnn · long · term”
State History Benchmarks Tried Checklist Preprints Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
5 papers in 2023–24 and 6 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 0.8% to 0.6%.
No longer reported by papers here: Penn TreeBank test (perplexity, last in 2019).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Siamese Recurrent Architectures for Learning Sentence Similarity (AAAI 2016 · 931 citations)
- 2017 · Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks (ICLR 2017 · 213 citations)
- 2018 · Targeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM (AAAI 2018 · 611 citations)
- 2019 · Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks (NeurIPS 2019 · 92 citations)
- 2020 · High-Performance Long-Term Tracking With Meta-Updater (CVPR 2020 · 293 citations)
- 2021 · Multi-timescale Representation Learning in LSTM Language Models (ICLR 2021 · 7 citations)
- 2022 · Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning (NeurIPS 2022 · 5 citations)
- 2023 · Embedding a Long Short-Term Memory Network in a Constraint Programming Framework for Tomato Greenhouse Optimisation (AAAI 2023 · 4 citations)
- 2024 · Parallelized Spatiotemporal Slot Binding for Videos (ICML 2024 · 0 citations)
- 2025 · Extended LSTMs for Knowledge Tracing: Peeking Inside the Black Box (Student Abstract) (AAAI 2025 · 1 citation)
- 2026 · Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks (NeurIPS 2026 · 0 citations)
Earliest papers here
- Siamese Recurrent Architectures for Learning Sentence Similarity (AAAI 2016 · 931 citations)
- Learning Statistical Scripts with LSTM Recurrent Neural Networks (AAAI 2016 · 179 citations)
- Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences (AAAI 2016 · 138 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Penn TreeBank test · perplexity (lower is better)
4 papers, 2018–2019 · not reported since 2019
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 52.8 | Regularizing and Optimizing LSTM Language Models (ICLR 2018 · 454 citations) |
What has been tried
50 papers here report 539 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 9%, hurt 23%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Increase alpha from the default while holding beta at 0 | grayscale sCIFAR-10 · accuracy | Helped | Tuning Frequency Bias of State Space Models (ICLR 2025 · 1 citation) | |
| Initialize slots by randomly sampling them from a learned Gaussian instead of using learned parameters | 2D and 3D experiments · overall performance | Hurt | Parallelized Spatiotemporal Slot Binding for Videos (ICML 2024 · 0 citations) | |
| Use Best rather than exponential parameterization | MNIST, LR 5e-6 · test loss | 2.241486 -> 2.241217 | Helped | StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization (ICML 2024 · 0 citations) |
| Add working memory | Sorting, 48K · accuracy | 19.04 -> 34.32 | Helped | Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture (ICML 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 6 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- sCIFAR-10 · 2 of 6 papers
- Long-Range Arena · 2 of 6 papers
Baselines
- S4 · 2 of 6 papers
- S4D · 2 of 6 papers
- S5 · 2 of 6 papers
- Liquid S4 · 2 of 6 papers
- Spectral SSM · 2 of 6 papers
- DSS · 2 of 6 papers
Recent preprints
Loading recent arXiv preprints…
Use it
Download BibTeX of 29 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- RNN training dynamics and gradients (102 papers)
- Brain-inspired recurrent architectures (87 papers)
- Efficient recurrent architectures (60 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Layer-wise analysis of language models (145 papers)
- Dataset condensation and flow estimation (126 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).