Research map / Architectures and efficiency / Attention and transformers
Long-context KV and memory attention
Speeds long-context LLM inference with hierarchical sparse attention and key-value memory designs.
88 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Attention and transformers, keyword cluster “kv · cache · long context”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
16 papers in 2023–24 and 39 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 2.7% to 3.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING (ICLR 2017 · 1,630 citations)
- 2018 · QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (ICLR 2018 · 411 citations)
- 2019 · Gaussian Transformer: A Lightweight Approach for Natural Language Inference (AAAI 2019)
- 2020 · MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (NeurIPS 2020 · 633 citations)
- 2021 · DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition (AAAI 2021 · 227 citations)
- 2022 · Squeezeformer: An Efficient Transformer for Automatic Speech Recognition (NeurIPS 2022 · 72 citations)
- 2023 · SmartAssign: Learning a Smart Knowledge Assignment Strategy for Deraining and Desnowing (CVPR 2023 · 30 citations)
- 2024 · Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs (CVPR 2024 · 15 citations)
- 2025 · Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · 15 citations)
- 2026 · The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path (ICML 2026 · 0 citations)
Earliest papers here
- A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING (ICLR 2017 · 1,630 citations)
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (ICLR 2018 · 411 citations)
- Deep Semantic Role Labeling With Self-Attention (AAAI 2018 · 335 citations)
What has been tried
64 papers here report 721 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 8%, hurt 24%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Move the gate from SDPA output to value projection | 400B tokens · Avg PPL | 5.761 -> 5.820 | Hurt | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · 15 citations) |
| Replace low-rank attention approximation with deformable attention | ARC-Easy, ARC-Challenge, OpenBookQA, PIQA · accuracy | Baseline/Ours/Deformable: arc-easy 73.4/73.2/24.4; arc-challenge 45.6/47.3/21.4; openbookqa 32.2/32.6/23.0; piqa 79.4/79 | Hurt | Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs (CVPR 2024 · 15 citations) |
| Add multiplicative attention alongside additive attention | EP and MI · accuracy and MSE | Helped | Arithmetic Feature Interaction Is Necessary for Deep Tabular Learning (AAAI 2024 · 13 citations) | |
| add the first input adapter | tabular benchmarks · task performance | Helped | TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · 5 citations) | |
| Use page size 16 instead of page size 32 | LongBench tasks, especially tight cache budgets · F1 / Rouge-L / accuracy | Mixed | ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction (NeurIPS 2024 · 5 citations) | |
| Permute the board tile order | Total average vs UCT 1× · win rate | 75% -> 64% | Hurt | Fast and Knowledge-Free Deep Learning for General Game Playing (Student Abstract) (AAAI 2024 · 1 citation) |
| add conditional LoRA for compression tokens instead of default unconditional LoRA | MetaICL, LaMP, and DailyDialog · accuracy (%) on MetaICL and LaMP; perplexity on DailyDialog | 69.4 -> 70.0; 66.3 -> 69.6; 83.9 -> 84.7; 82.6 -> 83.9; 6.01 -> 5.96; 6.42 -> 6.33 | Helped | Compressed Context Memory for Online Language Model Interaction (ICLR 2024 · 1 citation) |
| add random sparse attention selection to BaseConv | Pile validation, 162M–168M hybrids · log perplexity | 12.90 -> 12.13 | Helped | Zoology: Measuring and Improving Recall in Efficient Language Models (ICLR 2024 · 1 citation) |
| use CodeHunter instead of CodeHunter(AST) without AST parsing | Nexgen · Top-1 accuracy | 89.21% -> 93.86% | Helped | BERT-Based Code Learning for Exception Localization and Type Prediction (AAAI 2025 · 0 citations) |
| add Full Tree Attention | MT-Bench, temperature 0, Vicuna-7B · speedup | 3.10× -> 3.15× | Helped | Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding (ICML 2025 · 0 citations) |
| Replace softmax attention with non-causal linear attention | 57 real tabular datasets · Accuracy and runtime | Helped | TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · 0 citations) | |
| Reduce the adaptive-head ratio to 0.1 | RULER-128K · Accuracy | No clear effect | CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 17 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- RedPajama · 2 of 17 papers
Baselines
- XGBoost · 2 of 17 papers
- CatBoost · 2 of 17 papers
- LightGBM · 2 of 17 papers
- SAINT · 2 of 17 papers
- FTTransformer · 2 of 17 papers
- StreamingLLM · 2 of 17 papers
- SnapKV · 2 of 17 papers
- DuoAttention · 2 of 17 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 55 papers; from OpenAlex and the papers' affiliations).
- Alibaba Group · 4 papers
- Huawei · 3 papers
- Tencent · 3 papers
- Tsinghua University · 3 papers
- Peking University · 3 papers
- Shanghai Jiao Tong University · 2 papers
- Hong Kong University of Science and Technology · 2 papers
- Seoul National University · 2 papers
- Stanford University · 2 papers
- Meta · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Sparse and structured attention (163 papers)
- Attention head and block redesign (127 papers)
- Efficient image transformers (84 papers)
- Transformer neural operators for PDEs (69 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- State space and recurrent sequence models (172 papers)
- Sparse attention for long-context LLMs (147 papers)
- Efficient and robust vision transformers (124 papers)
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
- Hybrid and interpretable ViT designs (94 papers)
- KV cache eviction and compression (92 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).