Research map / Large language models / Reasoning and chain of thought

Efficient numeracy in language models through single-token number embeddings

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows frontier LLMs need thousands of reasoning tokens for multi-digit arithmetic, then proposes BitTokens, a single-token IEEE 754 bit-vector number encoding that lets small nanoGPT-2 models learn near-perfect arithmetic.

Problem

Frontier LLMs rely on long reasoning chains and multi-token number splitting for arithmetic, which is token-inefficient and limits the length of computations they can solve. Existing single-token number encodings (xVal, FoNE) fail several desiderata for learnable arithmetic.

Approach

Encode each number as one token by adding a learned [NUM] embedding plus a 64-bit IEEE 754 float64 bit vector (concatenated with the bit vector of its reciprocal), predicted by a sigmoid number head trained with bitwise BCE, so that sign, exponent and significand bits are directly accessible to the network for learning arithmetic algorithms.

Key results

• Frontier models with maximal reasoning reach high log-sMAPE on multiplication/division, but non-reasoning models stay below 60% log-sMAPE on multiplication, division, exponentiation and standard deviation; reasoning models use roughly 5k-30k tokens per calculation.
• BitTokens achieves near-perfect performance on comparison and single-step calculation tasks in multi-task training, outperforming single-digit, triple-digit, xVal and FoNE encodings.
• Ablations show base-2 encoding beats base-10 and appending the reciprocal and curriculum learning both improve performance; for the mean task single-digit and subword tokenizers outperform single-token methods.

Surprising findings

• For the mean task, multi-token (single-digit and subword) tokenizers outperform single-token methods, attributed to iterative digit-by-digit generation.
• Even GPT 5 with maximal reasoning reaches only about 34% exact match on standard deviation and 47% on exponentiation log-sMAPE-wise lower than on basic arithmetic.

Limitations

• Evaluated only on small (117M) nanoGPT-2 models trained from scratch, not production LLMs.
• Frontier-model results are upper bounds since tool use cannot be fully excluded; frontier evaluation uses a 500-sample subset.
• Does not yet handle number parsing for scientific notation or integration into text-based LLM pipelines.

Experimental setup

Datasets: FineWeb 10B custom numeracy dataset (9 tasks, 30M train / 10k val / 10k test per task)

Benchmarks: custom numeracy benchmark (MinMax, Interval, Sorting, Add, Mult, Div, Mean, Std, Exp) FineWeb validation perplexity

Metrics: log-sMAPE exact match accuracy perplexity tokens per problem

Baselines: single-digit tokenization triple-digit subword tokenization xVal FoNE GPT 5 Gemini 2.5 Pro DeepSeek v3.1 Qwen3 235B 2507

Builds on: xVal FoNE nanoGPT Muon

Closest papers by meaning

• Regress, Don't Guess: A Regression-like Loss on Number Tokens for Language Models (ICML 2025 · similarity 0.90)
• Number Cookbook: Number Understanding of Language Models and How to Improve It (ICLR 2025 · similarity 0.89)
• Interpreting and Improving Large Language Models in Arithmetic Calculation (ICML 2024 · similarity 0.88)
• OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step (NeurIPS 2024 · similarity 0.88)
• Transformers Can Do Arithmetic with the Right Embeddings (NeurIPS 2024 · similarity 0.87)
• Auto-Regressive Next-Token Predictors are Universal Learners (ICML 2024 · similarity 0.86)