Research map / Architectures and efficiency / Attention and transformers
Repeat After Me: Transformers are Better than State Space Models at Copying
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that transformers copy and retrieve from context far better than fixed-state generalized state space models (GSSMs such as Mamba and LSTM), using a depth-2 hash-based copy construction, synthetic training, and pretrained Pythia vs Mamba comparisons.
Problem
Whether fixed-size-state sequence models (GSSMs like SSMs, RNNs, linear attention) can match transformers at tasks requiring copying and retrieving arbitrary parts of the input context.
Approach
Proves a depth-2 transformer can copy exponentially long strings via hash-based n-gram lookup (with Hard-ALiBi positional encoding), while any GSSM with state size below L log D fails to copy uniform strings of length L; then validates the gap empirically in trained-from-scratch and pretrained models.
Key results
- Theory: a depth-2 transformer copies sequences with error below repeated-n-gram probability, enabling copying of length exponential in head count; any GSSM with state below L·log(D)−1 bits has error above 1/2 on uniform copy.
- Trained from scratch, transformers need about 100x fewer samples than the best GSSMs to learn copying length-300 strings; GSSMs collapse to near zero beyond length 50 training range while Hard-ALiBi transformer generalizes to length 1000.
- Pretrained Pythia (410M) outperforms Mamba 2.8B on phone-book lookup for phone-book sizes L>=70 and on copying natural-language strings from C4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a transformer instead of a GSSM for copying | Models trained on the copy task · Copy strings of length 300 | training samples required: 100x less samples | Helped |
| Use a transformer instead of a GSSM | Models trained on the copy task · Length generalization from strings ≤ 50 to strings up to 1000 | string-level accuracy | Helped |
| Use Hard-ALiBi positional encoding (ablation) | Transformer trained on the copy task · Length generalization from strings ≤ 50 to strings up to 1000 | string-level accuracy | Helped |
| Use ALiBi or NoPE instead of RoPE (ablation) | Transformer trained on the copy task · Longer copy sequences | string-level accuracy | Helped |
| Introduce duplicate n-grams of 5 or more tokens | Hard-ALiBi transformer · Copy task with duplicated n-grams | string-level accuracy | Hurt |
| Use suffix keys rather than prefix keys (ablation) | GSSMs on n-gram lookup · Length generalization on n-gram lookup | performance | Hurt |
| Use prefix keys rather than suffix keys (ablation) | GSSMs on n-gram lookup · Length generalization on n-gram lookup | performance | Helped |
| Use GSSMs rather than NoPE or ALiBi transformers | N-gram lookup with prefix keys · Length generalization on n-gram lookup | performance | Helped |
| Use a transformer instead of a GSSM | Pretrained models · Copy natural-language strings from C4 | string-level accuracy | Helped |
| Shuffle the word order of the input strings (ablation) | Pretrained Pythia and Mamba models · Copy strings from C4 | string-level accuracy | Hurt |
| Use a transformer instead of a GSSM | Pretrained models · Copy shuffled strings of length 300 | string-level accuracy: zero accuracy | Helped |
| Use a transformer instead of a GSSM | Pretrained models · Phone-book lookup | accuracy | Helped |
Surprising findings
- On prefix-key n-gram lookup, GSSMs match or beat NoPE and ALiBi transformers, though not Hard-ALiBi.
- Mamba achieves lower perplexity than Pythia at the same size yet performs far worse on copying and retrieval.
Limitations
- Theory only covers uniform copy distributions and representational capacity, not learnability in general.
- Pretrained comparisons cover only Pythia vs Mamba at limited sizes, with small evaluation sets (e.g., 50 SQuAD questions per bin).
Experimental setup
Datasets: C4 SQuAD The Pile (pretraining of Pythia and Mamba) synthetic copy and n-gram lookup data
Benchmarks: SQuAD synthetic copy task n-gram lookup task phone-book lookup C4 copy
Metrics: string-level accuracy character-level accuracy F1
Baselines: Mamba LSTM Pythia RoPE transformer NoPE transformer ALiBi transformer
Builds on: Mamba ALiBi Pythia induction heads FlashAttention
Closest papers by meaning
- Effective Context in Transformers: An Analysis of Fragmentation and Tokenization (NeurIPS 2026 · similarity 0.89)
- Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D (NeurIPS 2026 · similarity 0.88)
- The Expressive Capacity of State Space Models: A Formal Language Perspective (NeurIPS 2024 · similarity 0.88)
- On the Expressiveness and Length Generalization of Selective State Space Models on Regular Languages (AAAI 2025 · similarity 0.88)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.88)
- (How) Do Language Models Track State? (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.