Research map / Architectures and efficiency / Attention and transformers
Extrapolation by Association: Length Generalization Transfer In Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that length generalization can transfer from a longer-trained auxiliary task to a shorter-trained main task in transformers (from scratch and pretrained SmolLM-360M), and links transfer to reuse of attention heads/circuits.
Problem
How transformers extrapolate to longer inputs than seen in training; the paper asks whether length generalization on one task can be transferred from a related auxiliary task trained on longer inputs.
Approach
Length generalization emerges via task association: co-training a main task on short inputs with a related auxiliary task on longer inputs lets the main task extrapolate to the auxiliary's length range, a transfer not seen when training the main task alone; the effect correlates with shared attention-head usage.
Key results
- Co-training reverse add (main, length 16) with no carry and carry only (length 32) lets the main task extrapolate toward the auxiliary's range, while reverse add alone fails beyond length 16.
- Control pair reverse add with copy-first-op shows no length generalization transfer, supporting that task relatedness matters.
- Finetuning SmolLM-360M checkpoints shows transfer improves steadily with more pretraining (step 160K to 2.56M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Co-train reverse add with no carry and carry only | From-scratch transformer · Reverse add, lengths beyond training length | exact match accuracy | Helped |
| Co-train reverse add with reverse subtract | From-scratch transformer · Reverse add, lengths beyond training length | exact match accuracy | Helped |
| Co-train reverse add with n × 3 CoT multiply | From-scratch transformer · Reverse add, lengths beyond training length | exact match accuracy | Helped |
| Co-train string copy with MQAR | From-scratch transformer · String copy, lengths beyond training length | exact match accuracy | Helped |
| Co-train capitalize-reverse with capitalize and reverse | From-scratch transformer · Capitalize-reverse, lengths beyond training length | exact match accuracy | Helped |
| Co-train DFS trace with shortest path | From-scratch transformer · DFS trace, longer mazes | exact match accuracy | Helped |
| Co-train shortest path with DFS trace | From-scratch transformer · Shortest path, longer mazes | exact match accuracy | Helped |
| Use unrelated copy-first-op as an auxiliary task | Reverse add · Reverse add, lengths beyond training length | exact match accuracy | No clear effect |
| Use reverse as an auxiliary task | String copy · String copy, lengths beyond training length | exact match accuracy | No clear effect |
| Increase natural-language pretraining progress | SmolLM-360M finetuned on reverse add and shortest path · Reverse add and shortest path, out-of-distribution lengths | exact match accuracy | Helped |
| Use NoPE instead of RoPE (ablation) | From-scratch transformer · Reverse add, capitalize-reverse, string copy; transfer evaluation | generalization transfer | Hurt |
| Use only one auxiliary task instead of both no carry and carry only (ablation) | Reverse add · Reverse add, lengths beyond training length | exact match accuracy | Hurt |
Surprising findings
- Generalization transfer is unstable: different random seeds with the same dataset yield noticeably different outcomes.
- NoPE yields substantially weaker transfer than RoPE, contrary to the common view that NoPE is better for length generalization.
Limitations
- No formal theoretical framework explaining when and why transfer occurs.
- Experiments limited to simple synthetic algorithmic domains with deterministic solutions.
- Training instability: transfer succeeds only for some random seeds.
Experimental setup
Datasets: synthetic reverse add no carry carry only reverse subtract n x 3 CoT multiply string copy MQAR capitalize reverse capitalize-reverse DFS trace maze shortest path maze
Benchmarks: synthetic algorithmic task suites (arithmetic, string, maze) with length sweeps
Metrics: exact match accuracy generalization gap attention matrix difference head mean-ablation map difference
Baselines: main task trained alone NoPE model control tasks (copy-first-op, reverse)
Builds on: Llama RoPE multi-query associative recall Wilson's algorithm task hinting
Closest papers by meaning
- A Formal Framework for Understanding Length Generalization in Transformers (ICLR 2025 · similarity 0.91)
- What Algorithms can Transformers Learn? A Study in Length Generalization (ICLR 2024 · similarity 0.91)
- The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.90)
- Universal Length Generalization with Turing Programs (ICML 2025 · similarity 0.89)
- Exploring Transformer Extrapolation (AAAI 2024 · similarity 0.89)
- Understanding the Generalization of In-Context Learning in Transformers: An Empirical Study (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.