Research map / Architectures and efficiency / Attention and transformers
Transformers Can Do Arithmetic with the Right Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds Abacus Embeddings (per-digit positional embeddings shared across same-significance digits, with random offsets) to small transformers trained from scratch on arithmetic, combined with input injection and looped transformers, achieving length generalization to 100+ digit addition.
Problem
Transformers struggle to learn multi-digit arithmetic and to extrapolate to longer operands because they lose track of each digit's exact position within a number.
Approach
Learned positional embeddings assigned per digit according to its significance relative to the number start (with a random starting offset during training), so digits of equal significance share an embedding; combined with input injection and looped (recurrent) transformer blocks.
Key results
- Abacus Embeddings with input injection and looped transformer raise OOD addition accuracy from 92.9% to 99.1%, an 87% error reduction versus embeddings with standard architectures alone.
- Models trained on up to 20-digit operands reach up to 99% accuracy on 100-digit addition and generalize to 120-digit addition (6x length generalization vs. prior 2.5x).
- Abacus Embeddings with FIRE reach 68.63% OOD accuracy on sorting with 30-digit numbers vs. 55.32% for FIRE alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Abacus Embeddings and input injection with a looped transformer | Addition model trained on operands up to 20 digits · Addition, OOD | accuracy: 92.9% to 99.1% | Helped |
| Add Abacus Embeddings | Standard transformer · Addition, OOD length generalization | exact match accuracy | Helped |
| Use Abacus Embeddings instead of FIRE or NoPE | Addition transformer · Addition | exact match accuracy | Helped |
| Combine Abacus Embeddings with FIRE | 8-layer standard transformer · Sorting, all OOD (30×30) | exact match accuracy: 3.73; 2.65; 4.48 | Helped |
| Combine Abacus Embeddings with FIRE | 8-layer standard transformer · Sorting, OOD (number length - 30) | exact match accuracy: 55.32; 68.63; 67.28 | Helped |
| Combine Abacus Embeddings with FIRE | 8-layer standard transformer · Sorting, OOD (array length - 30) | exact match accuracy: 21.35; 9.67; 21.11 | Mixed |
| Use a looped transformer instead of a standard transformer | Sorting model · All OOD, minimum element only | accuracy: 49.73; 60.09; 68.51 | Helped |
| Use a looped transformer instead of a standard transformer | Sorting model · All OOD, exact string match | accuracy: 4.48; 3.84; 2.60 | Hurt |
| Use an 8-layer recurrent block with 2 recurrences instead of a non-recurrent 16-layer model (ablation) | Addition model with Abacus Embeddings · Addition, OOD | error rate: half the error | Helped |
| Remove loss masking before the equals sign (ablation) | Addition model · Addition | accuracy | Hurt |
| Remove input injection within recurrent-block layers, retaining it only at the first recurrent layer (ablation) | Addition looped transformer · Addition | performance | No clear effect |
| Divide recurrent-block gradients by the number of recurrences (ablation) | Addition looped transformer · Addition | performance | No clear effect |
Surprising findings
- Looped transformers without Abacus Embeddings outperform standard transformers with FIRE or NoPE embeddings on addition, and the looped model reaches best OOD performance with about 10x fewer parameters.
- Abacus Embeddings alone do not improve sorting exact-match accuracy over FIRE; the combination is needed for best all-OOD results, and the looped transformer is best at minimum-element identification but worst at full sorting.
Limitations
- Only small transformers trained from scratch; no natural language tasks evaluated.
- Abacus Embeddings are absolute embeddings, so generalization beyond the trained offset range k requires increasing k at test time.
- Multiplication and sorting results are limited to small scale due to compute cost; sorting exact-match gains are modest.
Experimental setup
Datasets: synthetic addition data (up to 20-digit operands) synthetic multiplication data (up to 15 digits) synthetic array sorting data synthetic bitwise OR data
Benchmarks: addition exact match accuracy (in-distribution, OOD up to 100 digits, 100+ digit OOD up to 160) multiplication up to 15 digits array sorting OOD (30-digit numbers, 30-element arrays)
Metrics: exact match accuracy
Baselines: FIRE NoPE RoPE Index hints (Zhou et al.) Standard transformer with input injection
Builds on: FIRE positional embeddings NoPE randomized positional embeddings progressive loss index hints
Closest papers by meaning
- Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure (NeurIPS 2024 · similarity 0.91)
- Understanding Addition in Transformers (ICLR 2024 · similarity 0.88)
- Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count (ICLR 2025 · similarity 0.88)
- Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning (ICLR 2025 · similarity 0.87)
- Efficient numeracy in language models through single-token number embeddings (ICML 2026 · similarity 0.87)
- Algorithmic Capabilities of Random Transformers (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.