Research map / Architectures and efficiency / Attention and transformers
Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Combines task-specific scratchpads with multi-level Position Coupling in decoder-only Transformers to achieve length generalization over both operand length and count for multi-operand addition and multiplication, with a 1-layer theoretical construction.
Problem
Transformers fail to length-generalize on arithmetic tasks where both the operand length and the number of operands increase at test time, notably multi-operand addition and multiplication with both operands variable in length.
Approach
Pair a cumulative-sum scratchpad (two-stage for multiplication) that keeps the number of attended tokens constant per step with multi-level Position Coupling (bi-level for addition, tri-level for multiplication) that tells the model which positions to attend to.
Key results
- Bi-level Position Coupling with scratchpad keeps median exact-match >= 90.0% on addition with up to 30 operands of length up to 30, trained on max 10 digits and 10 operands.
- NoPE or FIRE with scratchpad fail at 13 operands of length 13.
- Tri-level Position Coupling with two-stage scratchpad multiplies up to 20-digit by 15-digit integers with median exact match >= 78.55%, trained on up to 10 digits per operand.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine a scratchpad with bi-level Position Coupling | Decoder-only Transformer for multi-operand addition · Multi-operand addition, operands up to 30 digits and 30 operands | median exact-match accuracy | Helped |
| Combine a two-stage scratchpad with tri-level Position Coupling | Decoder-only Transformer for multiplication · Multiplication with both operand lengths varying | median exact-match accuracy | Helped |
| Add a scratchpad to Position Coupling | Transformer for parity · Parity, queries up to 100 bits | accuracy | Helped |
| Add a scratchpad to NoPE or FIRE (ablation) | Transformer for parity · Parity, queries up to 35 bits | accuracy | Helped |
| Add a scratchpad to NoPE or FIRE (ablation) | Transformer for multi-operand addition · Multi-operand addition | exact-match accuracy | Helped |
| Use NoPE or FIRE instead of Position Coupling with the scratchpad (ablation) | Transformer for multi-operand addition · 13 operands, each 12 digits long | exact-match accuracy | Hurt |
| Remove the scratchpad (ablation) | Transformer for multi-operand addition · Out-of-distribution operand lengths and counts | exact-match accuracy | Hurt |
| Train on a wider range of operand lengths and counts (ablation) | Transformer for multi-operand addition · Multi-operand addition | exact-match accuracy | Helped |
| Remove zero-padding from query and response (ablation) | Scratchpad + Position Coupling for multi-operand addition · Multi-operand addition | exact-match accuracy | Hurt |
| Use a 1-layer 2-head architecture (ablation) | Scratchpad + Position Coupling for multi-operand addition · Longer operands and varying operand counts | exact-match accuracy | Hurt |
| Use a 2-layer 2-head architecture instead of the 6-layer 8-head baseline (ablation) | Scratchpad + Position Coupling for multi-operand addition · All combinations of operand length 1–30 and count 2–30 | median exact-match accuracy: 67.8% or above -> at least 90.0% | Helped |
| Increase dimension per head from 128 to 256 (ablation) | 1-layer 8-head Transformer for multi-operand addition · Multi-operand addition | exact-match accuracy | Helped |
Surprising findings
- A 1-layer 4-head model outperforms the 6-layer 8-head baseline on addition, and 2-layer 2-head models reach >=90% across the grid.
- Position Coupling without scratchpad shows weak generalization for multiplication when one operand is short, contrary to the near-zero generalization reported by a related prior work's setup.
Limitations
- Approach only applies to tasks whose structure is well-defined and encodable by scratchpad and Position Coupling.
- Design choices rely heavily on task-specific intuition.
- Zero-padding aids length generalization but is not strictly necessary; performance degrades without it.
Experimental setup
Datasets: synthetic multi-operand addition S_A(n,m) synthetic multiplication S_M(n,m) synthetic parity
Benchmarks: multi-operand addition test sets T_A(n,m) up to 30 operands x 30 digits multiplication test sets T_M(n,m) up to 30x30 digits parity up to 100 bits
Metrics: exact-match accuracy median accuracy over seeds
Baselines: NoPE FIRE RoPE Position Coupling without scratchpad
Builds on: Position Coupling scratchpad FIRE NoPE
Closest papers by meaning
- A Formal Framework for Understanding Length Generalization in Transformers (ICLR 2025 · similarity 0.93)
- Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure (NeurIPS 2024 · similarity 0.92)
- Looped Transformers for Length Generalization (ICLR 2025 · similarity 0.92)
- What Algorithms can Transformers Learn? A Study in Length Generalization (ICLR 2024 · similarity 0.91)
- Universal Length Generalization with Turing Programs (ICML 2025 · similarity 0.90)
- Extrapolation by Association: Length Generalization Transfer In Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.