Research map / Architectures and efficiency / Attention and transformers
Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tests whether length-generalization limits predicted by C-RASP[pos] theory for retrieval and copying tasks persist in pretrained LLMs, finding a directional (right/forward over left/backward) bias from pretraining that fine-tuning can remove but the uniqueness bias persists.
Problem
Whether theoretical length-generalization limits of transformers on retrieval and copying tasks still matter for large-scale pretrained LLMs, or whether pretraining and scale overcome them.
Approach
Derives C-RASP[pos] expressibility predictions for unique/non-unique retrieval and copying tasks in left/right and forward/backward variants, then tests pretrained LLMs in-context, finds a directional asymmetry attributed to induction vs anti-induction circuit strength, and shows task-specific fine-tuning of GPT-2 1.5B removes the directional bias but not the uniqueness bias.
Key results
- Pretrained Llama-3 and Qwen2.5 models show a consistent directional bias: rightward/forward retrieval and copying outperform leftward/backward across lengths up to 100 tokens.
- After task-specific fine-tuning of GPT-2 1.5B, C-RASP[pos]-expressible tasks retain perfect accuracy out-of-distribution (lengths 101-200) while non-expressible non-unique tasks drop, and the directional bias disappears.
- Across Lorem Ipsum copying, transition indices were almost always at ambiguous tokens, indicating glitches arise from non-unique copying.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Retrieve to the right rather than to the left of the query | Pretrained Llama 3 and Qwen2.5 · Synthetic retrieval | accuracy | Helped |
| Copy forward rather than backward | Pretrained Llama 3 and Qwen2.5 · Synthetic copying | accuracy | Helped |
| Use unique-token rather than non-unique retrieval tasks | Pretrained Llama 3 and Qwen2.5 · Synthetic retrieval | accuracy | Helped |
| Copy repeated tokens forward rather than retrieve repeated-token answers | Pretrained Llama 3 and Qwen2.5 · Synthetic non-unique tasks | accuracy | Helped |
| Fine-tune on the target retrieval or unique-copying task | GPT-2 1.5B · Out-of-distribution length generalization | accuracy: 100% | Helped |
| Fine-tune on non-length-generalizable NLLast, NRLast, NF, or NB | GPT-2 1.5B · Out-of-distribution length generalization | accuracy | Hurt |
| Fine-tune pretrained models on the task | GPT-2 1.5B · Directional bias across retrieval and copying | accuracy | Helped |
| Copy Lorem Ipsum paragraphs containing ambiguous transitions | Pretrained model variants · Lorem Ipsum copying | transition-index ambiguity | Hurt |
| Reverse the required commit order | Llama 3 · Git commit history manipulation | accuracy | Hurt |
| Remove induction heads (ablation) | Pretrained and fine-tuned models · UF forward copying | accuracy: zero | Hurt |
| Remove anti-induction heads (ablation) | Pretrained and fine-tuned models · UB backward copying | accuracy: zero | Hurt |
| Remove anti-induction heads (ablation) | Pretrained and fine-tuned models · UF forward copying | accuracy | No clear effect |
Surprising findings
- Non-unique forward copying (NF) remains near perfect in pretrained models despite being theoretically non-length-generalizable.
- Pretrained models underperform on NRFirst, which theory predicts should be easier than NRLast for transformers trained from scratch.
Limitations
- Only two open model families evaluated; closed and recurrent models not covered.
- Fine-tuning experiments limited to a single 1.5B GPT-2 model due to resource limits.
- C-RASP[pos] length-generalization guarantee formally proven only for APE; RoPE results are empirical.
Experimental setup
Datasets: synthetic retrieval and copying strings Lorem Ipsum paragraphs git commit histories
Benchmarks: synthetic retrieval and copying tasks (UL, UR, NLFirst, NRFirst, NLLast, NRLast, UF, UB, NF, NB) Lorem Ipsum copying git revert/cherry-pick task
Metrics: accuracy transition-index ambiguity analysis
Baselines: transformers trained from scratch
Builds on: C-RASP[pos] Huang et al. length generalization framework induction heads retrieval heads
Closest papers by meaning
- From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers (NeurIPS 2025 · similarity 0.90)
- Transformer Layers as Painters (AAAI 2025 · similarity 0.89)
- Extrapolation by Association: Length Generalization Transfer In Transformers (NeurIPS 2025 · similarity 0.89)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.88)
- LLM Circuit Analyses Are Consistent Across Training and Scale (NeurIPS 2024 · similarity 0.88)
- Algorithmic Capabilities of Random Transformers (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.