Research map / Architectures and efficiency / Attention and transformers
Zoology: Measuring and Improving Recall in Efficient Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pretrains 17 attention and gated-convolution LMs on the Pile, traces their perplexity gap to associative recall, proposes multi-query associative recall (MQAR) as a synthetic proxy, and shows input-dependent sparse-attention hybrids of a BaseConv operator close most of the gap.
Problem
Gated-convolution language models trail attention on the Pile, and prior synthetic associative recall tests (single query, small vocabulary, fixed position) fail to explain or reproduce this gap.
Approach
Identify associative recall on real text as the main source of the gated-convolution gap, formalize a multi-query version (MQAR) that matches real-data behavior, show theoretically and empirically that fixed (input-independent) filters need model dimension scaling with sequence length, and close the gap by adding input-dependent sparse attention (programmatic or learned selection) to a minimal Bas
Key results
- Attention beats the best gated-convolution models by at least 2.14, 0.59 and 0.35 perplexity points at 70M, 160M and 360M scales; AR-hit tokens (6.4% of tokens) account for 82% of the average gap.
- A 70M attention model beats a 1.4B Hyena model on the AR-hit slice (2.41 vs 3.43 ppl).
- At 360M, BaseConv with 3 full-attention layers outperforms attention by 0.85 ppl, and programmatic sparse selection closes 85% of the AR-slice gap to attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace attention with gated-convolution sequence mixers | language models · Pile validation, 125M–169M models | log perplexity: 11.01 -> 11.60 (Hyena); 11.01 -> 11.64 (RWKV); 11.01 -> 12.06 (H3) | Hurt |
| replace attention with gated-convolution sequence mixers | language models · Pile validation, 351M–360M models | log perplexity: 9.44 -> 10.38 (H3); 9.44 -> 10.07 (Hyena); 9.44 -> 9.79 (RWKV) | Hurt |
| add random sparse attention selection to BaseConv (ablation) | BaseConv · Pile validation, 162M–168M hybrids | log perplexity: 12.90 -> 12.13 | Helped |
| add programmatic selection to BaseConv (ablation) | BaseConv · Pile validation, 162M–168M hybrids | log perplexity: 12.90 -> 11.06 | Helped |
| add learned selection to BaseConv (ablation) | BaseConv · Pile validation, 162M–168M hybrids | log perplexity: 12.90 -> 11.06 | Helped |
| add full attention to BaseConv (ablation) | BaseConv · Pile validation, 162M–168M hybrids | log perplexity: 12.90 -> 9.57 | Helped |
| add random sparse attention selection to BaseConv (ablation) | BaseConv · Pile validation, 351M–365M hybrids | log perplexity: 11.01 -> 12.94 | Hurt |
| add programmatic selection to BaseConv (ablation) | BaseConv · Pile validation, 351M–365M hybrids | log perplexity: 11.01 -> 9.54 | Helped |
| add learned selection to BaseConv (ablation) | BaseConv · Pile validation, 351M–365M hybrids | log perplexity: 11.01 -> 9.59 | Helped |
| add full attention to BaseConv (ablation) | BaseConv · Pile validation, 351M–365M hybrids | log perplexity: 11.01 -> 8.59 | Helped |
| use sparse attention selected programmatically at AR-hit tokens instead of random positions (ablation) | BaseConv hybrid · Pile validation, 351M–365M hybrids | log perplexity: 12.94 -> 9.54 | Helped |
| use learned sparse attention selection instead of random positions (ablation) | BaseConv hybrid · Pile validation, 351M–365M hybrids | log perplexity: 12.94 -> 9.59 | Helped |
Surprising findings
- A 70M attention model outperforms a 1.4B gated-convolution model on associative recall tokens despite being 20x smaller.
- Gated convolutions solve single-query synthetic associative recall perfectly yet fail on real text, motivating the multi-query formulation.
Experimental setup
Datasets: The Pile RedPajama (ArXiv, StackOverflow analysis) synthetic MQAR
Benchmarks: The Pile validation perplexity MQAR synthetic associative recall
Metrics: perplexity log-perplexity AR-hit perplexity slice fraction of perplexity gap closed synthetic accuracy
Baselines: Transformer attention (Llama-style) H3 Hyena RWKV Long Convolution Liquid S4 SeqBoat-style learned selection
Builds on: H3 Hyena RWKV LLaMA S4
Closest papers by meaning
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.89)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.89)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.88)
- On the Power of Convolution-Augmented Transformer (AAAI 2025 · similarity 0.87)
- The emergence of sparse attention: impact of data distribution and benefits of repetition (NeurIPS 2025 · similarity 0.87)
- Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.