AI Paper Atlas

Research map / Architectures and efficiency

Attention and transformers: research map

2,598 papers on Attention and transformers in Architectures and efficiency, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 6 clusters and 25 approaches. The busiest year so far is 2026.

What it is

The transformer, built on attention, is the standard architecture for language and increasingly vision. This topic studies how it works and how to make it efficient for long inputs: sparse and linear attention, compressing the KV cache, and alternatives such as state-space models (Mamba).

Terms you will see on this page:

Self-attention
each token computing a weighted mix of all other tokens.
KV cache
stored keys and values of past tokens, reused to generate the next ones.
State-space model (SSM)
a recurrent-style layer that handles long sequences in linear time.

Start reading with the most cited papers here:

Within Architectures and efficiency, its share grew from 29.8% in 2023–24 to 32.9% in 2025–26 (583 → 1,166 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).

2016: 8162017: 24172018: 34182019: 54192020: 108202021: 171212022: 254222023: 275232024: 420242025: 548252026: 70226

Explore Attention and transformers in the interactive map

Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.

What changed recently

Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):

Papers from 2025–2026 that are already cited most:

Approaches and key papers

Efficient attention and KV cache · 611 papers
Keywords: kv, cache, long context

Approaches in this cluster:

Most cited and most cited since 2024:

Self-attention design and theory · 550 papers
Keywords: self attention, attention mechanism, softmax

Approaches in this cluster:

Most cited and most cited since 2024:

Vision transformers and in-context learning · 391 papers
Keywords: vision, vit, transformers

Approaches in this cluster:

Most cited and most cited since 2024:

Long sequences and state-space models · 378 papers
Keywords: sequence, long, modeling

Approaches in this cluster:

Most cited and most cited since 2024:

Language transformer design · 359 papers
Keywords: transformers, language, icl

Approaches in this cluster:

Most cited and most cited since 2024:

Sparse attention for long context · 309 papers
Keywords: global, local, spatial

Approaches in this cluster:

Most cited and most cited since 2024:

Related topics in Architectures and efficiency

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy