Research map / Architectures and efficiency / RNNs and normalization
Teaching Arithmetic to Small Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that small transformers trained from scratch learn arithmetic much faster with reversed-output and chain-of-thought scratchpad data formats, with sample-efficiency gains linked to low-rank matrix completion.
Problem
How to efficiently teach small transformers trained from random initialization to perform arithmetic (addition, subtraction, multiplication, sine, square root) with next-token prediction, given that standard plain data formats learn poorly and slowly.
Approach
Reverse the output digit order so each output digit depends only on two operand digits and a carry, and add chain-of-thought scratchpads with intermediate steps; balance sampling by digit count and carry count. Phase transitions are linked to low-rank matrix completion of the addition table.
Key results
- Reverse-format 3-digit addition with NanoGPT reaches about 99.97% accuracy, versus about 87% for plain format, with a phase transition around 2500 training samples.
- Detailed scratchpad format achieves perfect 3-digit addition with about 1000 samples, while reverse requires more than double the samples of simplified scratchpad (2000).
- Joint training on five arithmetic tasks gives zero-shot multiplication accuracy of 14.28% versus 4.58% when trained individually.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train jointly on five arithmetic tasks instead of training on the task individually | NanoGPT · Multiplication | accuracy: 4.58 -> 14.28 | Helped |
| train jointly on five arithmetic tasks instead of training on the task individually | NanoGPT · Subtraction | accuracy: 79.97 -> 72.83 | Hurt |
| balance sampling by operand digit length (ablation) | NanoGPT addition training · 3-digit addition | overall accuracy: 97.74 -> 98.13 | Helped |
| balance the number of carry-on operations (ablation) | NanoGPT addition training · 3-digit addition | overall accuracy: 97.74 -> 98.29 | Helped |
| reverse the output digits | NanoGPT · 3-digit addition | accuracy: Plain plateaus at slightly over 85%; Reverse reaches 100% | Helped |
| add simplified scratchpad digit-sum and carry steps | NanoGPT addition training · 3-digit addition | training samples to reach 100% accuracy: 2000 samples | Helped |
| add detailed scratchpad intermediate steps | NanoGPT addition training · 3-digit addition | training samples to reach 100% accuracy: 1000 samples | Helped |
| use subtraction scratchpad Version 2 with operand comparison and swapping instead of Version 1 (ablation) | NanoGPT subtraction training · 3-digit subtraction | accuracy | Hurt |
| replace correct intermediate digit-sum and carry labels with random labels (ablation) | NanoGPT simplified-scratchpad addition training · Addition | sample efficiency | Hurt |
| exclude 500 operands from training (ablation) | NanoGPT reverse-format addition · 3-digit addition | overall accuracy: 99.97% -> 99.99% | No clear effect |
| exclude the least significant digit 5 during training (ablation) | NanoGPT reverse-format addition · 3-digit addition, excluded-digit examples | exclusion accuracy: 99.90% -> 66% | Hurt |
| add random perturbations to preceding output tokens (ablation) | NanoGPT plain-format addition · 3-digit addition | exact accuracy: 49.88% | Hurt |
Surprising findings
- Excluding up to 500 of the 3-digit numbers from training does not hurt addition accuracy, so the model generalizes beyond low-rank matrix completion.
- Reverse format is not particularly effective for multiplication, and scratchpad gives limited gains for sine and square root.
Limitations
- Length generalization beyond trained digit lengths is difficult.
- Scratchpad formats are verbose and less token-efficient; gains are limited for sine and square root.
- Experiments are on small models trained from scratch on character-level arithmetic with limited digit ranges.
Experimental setup
Datasets: Shakespeare corpus self-constructed arithmetic datasets (addition, subtraction, multiplication, sine, square root)
Benchmarks: self-constructed arithmetic test sets (addition, subtraction, multiplication, sine, square root)
Metrics: accuracy sample complexity exact accuracy relaxed accuracy
Baselines: plain format reverse format random sampling
Builds on: chain-of-thought prompting least-to-most prompting Scratchpads for intermediate computation
Closest papers by meaning
- Assign and Add: A Mechanistic Study of Compositional Arithmetic (NeurIPS 2026 · similarity 0.89)
- Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks (NeurIPS 2024 · similarity 0.88)
- Algorithmic Capabilities of Random Transformers (NeurIPS 2024 · similarity 0.88)
- Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count (ICLR 2025 · similarity 0.88)
- Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges (ICML 2025 · similarity 0.87)
- Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.