Research map / Architectures and efficiency / Attention and transformers
When can transformers reason with abstract symbols?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that transformers trained by gradient descent learn abstract relational templates and generalize to unseen symbols (regression), while MLPs fail, and proposes adding one trainable scalar per attention head that improves data efficiency.
Problem
Whether and when transformers can perform relational reasoning with abstract symbols, generalizing out-of-distribution to symbols never seen during training, and how to improve their data efficiency for this.
Approach
Formalizes relational reasoning as template tasks; shows via a random-features kernel analysis that one-layer transformers generalize to unseen symbols given enough data while MLPs cannot; proposes adding a trainable identity term (a·I to W_K W_Q^T, and b·I to W_V W_O^T) per head, adding only two parameters per head, to fix copying failures and improve data efficiency.
Key results
- Modified transformer (per-head identity scalings) lowers the data needed for template-task generalization by about an order of magnitude versus vanilla transformer.
- GPT-2 with two extra parameters per head (288 total on 117M model) improves Wikitext perplexity when trained from random initialization.
- Theory: MLPs trained by SGD or Adam fail to generalize to unseen symbols regardless of data size; standard transformers fail at copying unseen symbols at large embedding dimension.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a trainable identity scaling to each attention head's key-query matrix | Transformer · Template tasks | data efficiency: by an order of magnitude | Helped |
| Add trainable identity scalings to both key-query and value-output matrices | Transformer · Length-k majority task | test loss: by at least an order of magnitude | Helped |
| Add the proposed transformer modifications | GPT-2 · Wikitext | perplexity | Helped |
| Use pretrained rather than random initialization | GPT-2 · αβα vs. αββ template task | test loss | Helped |
| Increase the number of attention heads (ablation) | Transformer · αβα vs. αββ template task | test loss | Helped |
| Increase transformer depth (ablation) | Transformer · αβα vs. αββ template task, n = 512 | generalization | No clear effect |
| Increase transformer depth (ablation) | Transformer · αβα vs. αββ template task, n = 1024 | generalization | Helped |
| Increase embedding dimension (ablation) | Transformer · αβα vs. αββ template task, n = 1024 | generalization | Hurt |
| Reduce batch size (ablation) | Transformer · αβα vs. αββ template task | generalization | Helped |
| Train only the final unembedding layer instead of all parameters (ablation) | Transformer · αβα vs. αββ template task | data efficiency | Hurt |
| Increase attention-weight initialization magnitude (ablation) | Transformer · αβα vs. αββ template task | generalization | Helped |
| Increase random-template length (ablation) | Transformer · Random template task | test loss | Hurt |
Surprising findings
- Standard transformers fail to copy symbols unseen in training as embedding dimension grows (an inverse-scaling-like failure).
- Training only the final unembedding layer still yields out-of-distribution generalization, though it needs more samples and epochs.
Limitations
- Positive generalization theory is for one-layer transformers in the infinite-width random-features/kernel regime with trained final layer only.
- Copying theorem for next-token prediction is established only at early training time and for length-1 inputs.
- Bounds on required data and width are not tight; conjectured results for other activations are unproven.
Experimental setup
Datasets: Wikitext synthetic template task data
Benchmarks: Wikitext perplexity synthetic template tasks (alpha beta alpha vs alpha beta beta, majority task)
Metrics: test loss on unseen symbols perplexity
Baselines: MLP vanilla transformer GPT-2 random initialization MLP with XX^T augmentation
Builds on: neural tangent kernel GPT-2 kernel ridge regression
Closest papers by meaning
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.90)
- Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? (ICLR 2025 · similarity 0.90)
- Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalization (NeurIPS 2025 · similarity 0.90)
- Transformers Provably Solve Parity Efficiently with Chain of Thought (ICLR 2025 · similarity 0.89)
- When No Paths Lead to Rome: Benchmarking Systematic Neural Relational Reasoning (NeurIPS 2025 · similarity 0.89)
- Can Transformers Reason Logically? A Study in SAT Solving (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.