Research map / Architectures and efficiency / Attention and transformers
Language model pretraining and induction
Studies positional encodings, embeddings, and induction heads in language-model pretraining.
99 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Attention and transformers, keyword cluster “transformers · language · icl”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
16 papers in 2023–24 and 24 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 2.7% to 2.1%.
No longer reported by papers here: Penn Treebank test (perplexity, last in 2020).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations (AAAI 2016 · 254 citations)
- 2017 · Language Modeling with Gated Convolutional Networks (ICML 2017 · 1,114 citations)
- 2018 · Search Engine Guided Neural Machine Translation (AAAI 2018 · 152 citations)
- 2019 · What do you learn from context? Probing for sentence structure in contextualized word representations (ICLR 2019 · 135 citations)
- 2020 · ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (ICLR 2020 · 535 citations)
- 2021 · DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION (ICLR 2021 · 913 citations)
- 2022 · Selecting Optimal Context Sentences for Event-Event Relation Extraction (AAAI 2022 · 36 citations)
- 2023 · Learning to Compress Prompts with Gist Tokens (NeurIPS 2023 · 16 citations)
- 2024 · The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains (NeurIPS 2024 · 6 citations)
- 2025 · PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models (CVPR 2025 · 3 citations)
- 2026 · Progressive Cramming: Reliable Token Compression and What It Reveals (ICML 2026 · 0 citations)
Earliest papers here
- A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations (AAAI 2016 · 254 citations)
- Latent Attention For If-Then Program Synthesis (NeurIPS 2016 · 46 citations)
- Language Modeling with Gated Convolutional Networks (ICML 2017 · 1,114 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Penn Treebank test · perplexity (lower is better)
4 papers, 2017–2020 · not reported since 2020
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 70.9 | Pointer Sentinel Mixture Models (ICLR 2017 · 460 citations) |
| 2019 | 55.7 | Deep Residual Output Layers for Neural Language Generation (ICML 2019 · 4 citations) |
| 2020 | 54.2 | DeFINE: Deep Factorized Input Token Embeddings for Neural Sequence Modeling (ICLR 2020 · 13 citations) |
What has been tried
83 papers here report 911 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 8%, hurt 17%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use one transformer layer instead of two | ICL-MC · convergence to the Bayes-optimal solution | Hurt | The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains (NeurIPS 2024 · 6 citations) | |
| Use cluster centers as representatives instead of merging cluster tokens | multiple reduction ratios · benchmark performance | Hurt | PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models (CVPR 2025 · 3 citations) | |
| Replace ground-truth response prefixes with dummy tokens during training and inference | G_{2, 5} · exact-match accuracy | 48.9 -> 99.9 | Helped | The Pitfalls of Next-Token Prediction (ICML 2024 · 2 citations) |
| use dynamically sized, text-aligned patches instead of fixed-size patches | PG-19 · bits-per-byte | 1.112 -> 1.009 | Helped | SpaceByte: Towards Deleting Tokenization from Large Language Modeling (NeurIPS 2024 · 2 citations) |
| Add fuse-attention on the encoder side only | CoGnition CG-test · Instance-level CTER | -7.2% | Helped | Layer-Wise Representation Fusion for Compositional Generalization (AAAI 2024 · 1 citation) |
| splitting evaluation sequence into retrieval half and evaluation half (no overlap) | Pile all tasks · bits per byte | 82% -> 83% | No clear effect | Test-Time Training on Nearest Neighbors for Large Language Models (ICLR 2024 · 1 citation) |
| Scale tokenizer training data by byte premium | FLORES-200 · CTC | t(3544)=-0.615, p=0.539 | No clear effect | Explaining and Mitigating Crosslingual Tokenizer Inequities (NeurIPS 2025 · 0 citations) |
| Remove the top 10% of PTH outputs | Classification datasets · separability score | Hurt | Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning (NeurIPS 2025 · 0 citations) | |
| Use second-order OLA instead of first-order OLA | Dependency parsing from OLA, MLM · UAS | 81.29 -> 72.86 | Hurt | Order-Level Attention Similarity Across Language Models: A Latent Commonality (NeurIPS 2025 · 0 citations) |
| use continuous tokens instead of discrete tokens | CALVIN ABC · task performance | 12.7% | Hurt | BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learning (NeurIPS 2025 · 0 citations) |
| Add KV sharing to CoLM with chains {16, 16} and dim 2560 | Commonsense tasks (zero-shot) · Avg. acc_norm | 45.27 -> 44.80 | Hurt | Chain-of-Model Learning for Language Model (NeurIPS 2025 · 0 citations) |
| use a fixed projection subspace instead of learned, adaptive subspaces | FineWeb validation, 10K steps · perplexity | 22.64 -> 26.57 | Hurt | Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 13 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- IMDB · 2 of 13 papers
- FineWeb · 2 of 13 papers
- FineWeb-edu · 2 of 13 papers
Baselines
- zero-shot prompting · 2 of 13 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 40 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 2 papers
- Stanford University · 2 papers
- Inria · 2 papers
- Google · 2 papers
- Microsoft · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Transformer block and depth design (89 papers)
- Efficient language transformers (105 papers)
- Transformer expressivity and circuits (56 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- State space and recurrent sequence models (172 papers)
- Sparse and structured attention (163 papers)
- Sparse attention for long-context LLMs (147 papers)
- Attention head and block redesign (127 papers)
- Efficient and robust vision transformers (124 papers)
- Hybrid and interpretable ViT designs (94 papers)
- KV cache eviction and compression (92 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).