Research map / Architectures and efficiency / Attention and transformers
Sparse efficient transformers for long context
Reduces attention cost via sparse, hierarchical, or recurrent-memory designs for long sequences.
76 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Attention and transformers, keyword cluster “sequence · long · modeling”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
26 papers in 2023–24 and 28 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 4.5% to 2.4%.
No longer reported by papers here: LRA average (accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · Music Transformer: Generating Music with Long-Term Structure (ICLR 2019 · 305 citations)
- 2020 · Reformer: The Efficient Transformer (ICLR 2020 · 310 citations)
- 2021 · Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention (AAAI 2021 · 342 citations)
- 2022 · Transformer Quality in Linear Time (ICML 2022 · 44 citations)
- 2023 · Hyena Hierarchy: Towards Larger Convolutional Language Models (ICML 2023 · 74 citations)
- 2024 · FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (ICLR 2024 · 143 citations)
- 2025 · Vision-LSTM: xLSTM as Generic Vision Backbone (ICLR 2025 · 20 citations)
- 2026 · Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking (ICML 2026 · 0 citations)
Earliest papers here
- Music Transformer: Generating Music with Long-Term Structure (ICLR 2019 · 305 citations)
- Neural Speech Synthesis with Transformer Network (AAAI 2019)
- Reformer: The Efficient Transformer (ICLR 2020 · 310 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
LRA average · accuracy (higher is better)
4 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 62.9 | Long-Short Transformer: Efficient Transformers for Language and Vision (NeurIPS 2021 · 56 citations) |
What has been tried
57 papers here report 637 measured changes (a component added, removed or swapped, compared with a baseline): helped 66%, no clear effect 10%, hurt 20%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| use bidirectional instead of unidirectional traversal | ImageNet-1K and ADE20K · accuracy / mIoU | 72.2 -> 73.7; 28.6 -> 31.7 | Helped | Vision-LSTM: xLSTM as Generic Vision Backbone (ICLR 2025 · 20 citations) |
| Increase the number of training segments to 5 or more | Synthetic memorization, detection, and reasoning tasks · generalization to longer sequences | Helped | Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · 11 citations) | |
| Use L2 normalization instead of L1 normalization, with 1+ELU feature map | Zero-shot average, 340M params / 15B tokens · accuracy | 40.1 -> 42.1 | Helped | Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · 5 citations) |
| add contrastive loss to proxy-token training | LRA average · accuracy | 58.7 -> 58.9 | Helped | Proxyformer: Nyström-Based Linear Transformer with Trainable Proxy Tokens (AAAI 2024 · 3 citations) |
| Add end-to-end Stage 3 distillation after Stage 2 | Downstream average · accuracy | 53.3 -> 62.3 | Helped | Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · 3 citations) |
| Remove previous actions from the context | Meta-World and DMControl · normalized score | Helped | A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks (ICML 2025 · 2 citations) | |
| Add a one-layer convolutional patch embedding to the naive LMU | Speech Commands V2 · accuracy | 76.17 -> 89.28 | Helped | LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units (ICLR 2024 · 2 citations) |
| Increase projection dimension n | PG-19 sample · perplexity | Helped | Radar: Fast Long-Context Decoding for Any Transformer (ICLR 2025 · 1 citation) | |
| Remove the tensor-core Monarch decomposition while retaining fusion | FFT convolution, sequence length 1K · forward latency | 0.24 -> 0.67 ms | Hurt | FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores (ICLR 2024 · 1 citation) |
| Disable activation recomputation | 4k sequence, batch size 1, single A100 · TFLOPS | 2684.67 -> 3115.03 | Helped | Mini-Sequence Transformers: Optimizing Intermediate Memory for Long Sequences Training (NeurIPS 2024 · 1 citation) |
| Use min-max region metadata summarization instead of min-only summarization | Region-pair distance relationship preservation · Pearson correlation | 0.937 -> 0.975 | Helped | Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images (ICML 2025 · 0 citations) |
| Fuse propagation steps into one kernel | A100, 1024×1024, batch 16, 8 channels · forward runtime | 71.4 ms -> 57.4 ms; 1.2× | Helped | GSPN-2: Efficient Parallel Sequence Modeling (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 18 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- WikiText2 · 2 of 18 papers
- ImageNet-1K · 2 of 18 papers
- WikiText-103 · 2 of 18 papers
- PG-19 · 2 of 18 papers
- LongBench · 2 of 18 papers
Baselines
- Mamba · 6 of 18 papers
- StreamingLLM · 2 of 18 papers
- DeepSpeed-Ulysses · 2 of 18 papers
- Ring Attention · 2 of 18 papers
- ConvNeXt · 2 of 18 papers
- Reformer · 2 of 18 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 54 papers; from OpenAlex and the papers' affiliations).
- Princeton University · 4 papers
- Carnegie Mellon University · 4 papers
- Stanford University · 3 papers
- Google · 3 papers
- National University of Singapore · 3 papers
- Massachusetts Institute of Technology · 3 papers
- Covariant · 2 papers
- University of California, Berkeley · 2 papers
- University of Southern California · 2 papers
- Amazon · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- State space and recurrent sequence models (172 papers)
- Linear attention approximations (76 papers)
- Mamba architectures (40 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- Sparse and structured attention (163 papers)
- Sparse attention for long-context LLMs (147 papers)
- Attention head and block redesign (127 papers)
- Efficient and robust vision transformers (124 papers)
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
- Hybrid and interpretable ViT designs (94 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).