Research map / Architectures and efficiency / Attention and transformers
State space and recurrent sequence models
Builds SSM, long-convolution, and recurrent models for efficient long-sequence modeling.
172 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Attention and transformers, keyword cluster “sequence · long · modeling”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Few recent reports None of the tracked benchmarks has a report from 2025–26.
38 papers in 2023–24 and 93 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 6.5% to 8.0%.
No longer reported by papers here: LRA Image (accuracy, last in 2024); LRA ListOps (accuracy, last in 2024); LRA Pathfinder (accuracy, last in 2024); LRA Retrieval (accuracy, last in 2024); LRA Text (accuracy, last in 2024); LRA Path-X (accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (ICML 2016 · 628 citations)
- 2017 · Global Context-Aware Attention LSTM Networks for 3D Action Recognition (CVPR 2017 · 705 citations)
- 2018 · DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding (AAAI 2018 · 750 citations)
- 2019 · Character-Level Language Modeling with Deeper Self-Attention (AAAI 2019 · 303 citations)
- 2020 · Time-aware Large Kernel Convolutions (ICML 2020 · 21 citations)
- 2021 · MSA Transformer (ICML 2021 · 368 citations)
- 2022 · Efficiently Modeling Long Sequences with Structured State Spaces (ICLR 2022 · 498 citations)
- 2023 · HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (NeurIPS 2023 · 157 citations)
- 2024 · Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling (ICML 2024 · 54 citations)
- 2025 · Titans: Learning to Memorize at Test Time (NeurIPS 2025 · 18 citations)
- 2026 · Mixture of Horizons in Action Chunking (ICML 2026 · 0 citations)
Earliest papers here
- Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (ICML 2016 · 628 citations)
- An Online Sequence-to-Sequence Model Using Partial Conditioning (NeurIPS 2016 · 53 citations)
- Global Context-Aware Attention LSTM Networks for 3D Action Recognition (CVPR 2017 · 705 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
LRA Image · accuracy (higher is better)
7 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 88.65 | Efficiently Modeling Long Sequences with Structured State Spaces (ICLR 2022 · 498 citations) |
| 2023 | 89.98 | ChordMixer: A Scalable Neural Attention Model for Sequences with Different Length (ICLR 2023 · 4 citations) |
| 2024 | 91.12 | Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences (ICML 2024 · 0 citations) |
LRA ListOps · accuracy (higher is better)
6 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 60.6 | Diagonal State Spaces are as Effective as Structured State Spaces (NeurIPS 2022 · 64 citations) |
| 2023 | 62.15 | Simplified State Space Layers for Sequence Modeling (ICLR 2023 · 81 citations) |
LRA Pathfinder · accuracy (higher is better)
6 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 94.2 | Efficiently Modeling Long Sequences with Structured State Spaces (ICLR 2022 · 498 citations) |
| 2023 | 96.69 | ChordMixer: A Scalable Neural Attention Model for Sequences with Different Length (ICLR 2023 · 4 citations) |
LRA Retrieval · accuracy (higher is better)
6 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 90.9 | Efficiently Modeling Long Sequences with Structured State Spaces (ICLR 2022 · 498 citations) |
| 2023 | 91.4 | Simplified State Space Layers for Sequence Modeling (ICLR 2023 · 81 citations) |
| 2024 | 92.04 | State-Free Inference of State-Space Models: The *Transfer Function* Approach (ICML 2024 · 1 citation) |
LRA Text · accuracy (higher is better)
6 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 86.82 | Efficiently Modeling Long Sequences with Structured State Spaces (ICLR 2022 · 498 citations) |
| 2023 | 89.31 | Simplified State Space Layers for Sequence Modeling (ICLR 2023 · 81 citations) |
| 2024 | 91.1 | Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences (ICML 2024 · 0 citations) |
LRA Path-X · accuracy (higher is better)
5 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 96.35 | Efficiently Modeling Long Sequences with Structured State Spaces (ICLR 2022 · 498 citations) |
| 2023 | 98.58 | Simplified State Space Layers for Sequence Modeling (ICLR 2023 · 81 citations) |
What has been tried
108 papers here report 1,211 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 7%, hurt 22%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add RC equivariance | Genomics benchmark tasks overall · accuracy | Mixed | Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling (ICML 2024 · 54 citations) | |
| use 2048 local-structure tokens instead of no structure tokens | ProteinGYM · Spearman ρs | 0.392 -> 0.504 | Helped | ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention (NeurIPS 2024 · 37 citations) |
| Add attention using MAC | Long Context · acc | 92.68 -> 97.95 | Helped | Titans: Learning to Memorize at Test Time (NeurIPS 2025 · 18 citations) |
| Use Dynamic-TUP rather than Static-TUP | AMASS motion prediction · MPJPE (Avg.) | 24.6 -> 24.4 | Helped | Skeleton-in-Context: Unified Skeleton Sequence Modeling with In-Context Learning (CVPR 2024 · 17 citations) |
| improve the linear attention implementation | Pile · perplexity | 15.91 -> 15.23 | Helped | Learning to (Learn at Test Time): RNNs with Expressive Hidden States (ICML 2025 · 15 citations) |
| Add the Cognitive Feature Decomposition layer | AL2005 · AUC | Helped | Cognitive Fluctuations Enhanced Attention Network for Knowledge Tracing (AAAI 2025 · 11 citations) | |
| add sequence alignment and input-dependent parameterization to quasiseparable mixer | GLUE · GLUE average | 75.1 -> 79.7 | Helped | Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers (NeurIPS 2024 · 11 citations) |
| Use reverse-complement parameter sharing (PS) for DNA-xLSTM instead of the non-equivariant model with RC augmentation | DNA pre-training, MLM · cross-entropy loss | Mixed | Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences (ICLR 2025 · 8 citations) | |
| add trailerness encoding to the movie sequence | MAD · F1-score | 50.20 -> 52.38 | Helped | Towards Automated Movie Trailer Generation (CVPR 2024 · 8 citations) |
| Stack diagonal SSM recurrences | anti-symmetric signature prediction · validation accuracy | Helped | Theoretical Foundations of Deep Selective State-Space Models (NeurIPS 2024 · 8 citations) | |
| remove rotary relative position embedding | SlimPajama validation, 16384 · Perplexity | 9.57 -> 314.78 | Hurt | Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling (ICLR 2025 · 7 citations) |
| Use clustered triplet selection rather than 1 cluster | Uniprot · HR@1 | 48.30 -> 50.50 | Helped | Neural Embeddings for kNN Search in Biological Sequence (AAAI 2024 · 7 citations) |
What reviewers will expect
Datasets and baselines named in the 53 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- SlimPajama · 5 of 53 papers
- The Pile · 4 of 53 papers
- FineWeb-Edu · 4 of 53 papers
- LAMBADA · 4 of 53 papers
- PIQA · 4 of 53 papers
- HellaSwag · 4 of 53 papers
- FineWeb · 3 of 53 papers
- WinoGrande · 3 of 53 papers
Baselines
- Mamba · 14 of 53 papers
- Transformer++ · 14 of 53 papers
- Mamba2 · 11 of 53 papers
- DeltaNet · 10 of 53 papers
- GLA · 8 of 53 papers
- RetNet · 7 of 53 papers
- Gated DeltaNet · 6 of 53 papers
- H3 · 4 of 53 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 131 papers; from OpenAlex and the papers' affiliations).
- Stanford University · 12 papers
- Carnegie Mellon University · 9 papers
- Microsoft · 6 papers
- NVIDIA · 6 papers
- Princeton University · 5 papers
- Mila · 5 papers
- Meta · 5 papers
- Tel Aviv University · 4 papers
- State University of New York · 4 papers
- IBM · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Sparse efficient transformers for long context (76 papers)
- Linear attention approximations (76 papers)
- Mamba architectures (40 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- Sparse and structured attention (163 papers)
- Sparse attention for long-context LLMs (147 papers)
- Attention head and block redesign (127 papers)
- Efficient and robust vision transformers (124 papers)
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
- Hybrid and interpretable ViT designs (94 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).