Research map / Architectures and efficiency / Attention and transformers
Block-sparse attention kernels
Accelerates attention with block-sparse patterns and hardware-efficient kernels for LLMs and video generation.
60 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Attention and transformers, keyword cluster “global · local · spatial”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
13 papers in 2023–24 and 37 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 2.2% to 3.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin (NeurIPS 2017 · 73 citations)
- 2019 · Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry (CVPR 2019 · 110 citations)
- 2021 · Two-Stream Convolution Augmented Transformer for Human Activity Recognition (AAAI 2021 · 228 citations)
- 2022 · CSWin Transformer: A General Vision Transformer Backbone With Cross-Shaped Windows (CVPR 2022 · 1,316 citations)
- 2023 · Compact Transformer Tracker with Correlative Masked Modeling (AAAI 2023 · 104 citations)
- 2024 · FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (NeurIPS 2024 · 71 citations)
- 2025 · MobileMamba: Lightweight Multi-Receptive Visual Mamba Network (CVPR 2025 · 91 citations)
- 2026 · LUCID: Attention with Preconditioned Representations (ICML 2026 · 0 citations)
Earliest papers here
- Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin (NeurIPS 2017 · 73 citations)
- Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry (CVPR 2019 · 110 citations)
- Two-Stream Convolution Augmented Transformer for Human Activity Recognition (AAAI 2021 · 228 citations)
What has been tried
40 papers here report 478 measured changes (a component added, removed or swapped, compared with a baseline): helped 71%, no clear effect 5%, hurt 23%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add knowledge distillation (soft distillation from TResNet-L teacher) | ImageNet-1K · Top-1 | +2.0 | Helped | MobileMamba: Lightweight Multi-Receptive Visual Mamba Network (CVPR 2025 · 91 citations) |
| Add GEMM-softmax pipelining | Non-causal FP16 attention, batch 4, sequence length 8448, 16 heads, head dimension 128 · TFLOPs/s | 582 -> 661 | Helped | FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (NeurIPS 2024 · 71 citations) |
| Add self-attention to the CNN backbone | IXI registration · Dice | 0.785±0.081 -> 0.801±0.072 | Helped | H-ViT: A Hierarchical Vision Transformer for Deformable Image Registration (CVPR 2024 · 37 citations) |
| Add joint polyline encoding to the attention variant | Argoverse 2 validation set · minFDE6 | 1.093 -> 1.094 | No clear effect | Trajectory Mamba: Efficient Attention-Mamba Forecasting Model Based on Selective SSM (CVPR 2025 · 27 citations) |
| Add cross-modal correlation modulation | RGBT234 · PR | 87.5 -> 88.3 | Helped | Cross-modulated Attention Transformer for RGBT Tracking (AAAI 2025 · 26 citations) |
| use adaptive thermal diffusivity predicted by shared FVEs rather than fixed k = 0.0 | ImageNet-1K · top-1 accuracy (%) | 81.0 -> 82.2 | Helped | Building Vision Models upon Heat Conduction (CVPR 2025 · 25 citations) |
| add all three losses, Lpose, Lkt, and Lvt | MSCOCO val2017 · AP | 55.4% -> 72.3% | Helped | SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation (CVPR 2024 · 14 citations) |
| Add sparse 3D compression to the sparse 3D window module | Image-to-3D generation · mesh quality | No clear effect | Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention (NeurIPS 2025 · 9 citations) | |
| Vary the grouping number G | CIFAR-100 · Top-1 error | Mixed | EAN: An Efficient Attention Module Guided by Normalization for Deep Neural Networks (AAAI 2024 · 7 citations) | |
| add the proposed tokenization method | Tablet(1, 2) · RMSEP | 9.977 -> 8.491 | Helped | Analytical-Chemistry-Informed Transformer for Infrared Spectra Modeling (AAAI 2025 · 5 citations) |
| Add semantic-aware permutation | Attention recall across varying densities · attention recall | Helped | Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation (NeurIPS 2025 · 5 citations) | |
| replace global full attention with linear attention | BRACS tumor subtyping (20x) · F1 | 0.706 0.025 -> 0.693 0.024 | Hurt | Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis (NeurIPS 2024 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 20 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1K · 2 of 20 papers
- MS COCO 2017 · 2 of 20 papers
- ADE20K · 2 of 20 papers
- LongBench · 2 of 20 papers
- InfiniteBench · 2 of 20 papers
- RULER · 2 of 20 papers
Baselines
- MInference · 3 of 20 papers
- FlashAttention-2 · 3 of 20 papers
- FlexPrefill · 2 of 20 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 50 papers; from OpenAlex and the papers' affiliations).
- Huazhong University of Science and Technology · 3 papers
- Zhejiang University · 3 papers
- Stanford University · 3 papers
- Tsinghua University · 3 papers
- ETHZ - ETH Zurich · 3 papers
- Peking University · 2 papers
- Tencent · 2 papers
- Shanghai Jiao Tong University · 2 papers
- Harbin Institute of Technology · 2 papers
- The University of Hong Kong · 2 papers
Use it
Download BibTeX of 32 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Sparse attention for long-context LLMs (147 papers)
- KV cache eviction and compression (92 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- State space and recurrent sequence models (172 papers)
- Sparse and structured attention (163 papers)
- Attention head and block redesign (127 papers)
- Efficient and robust vision transformers (124 papers)
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
- Hybrid and interpretable ViT designs (94 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).