Research map / Large language models / Agents, code and math
Number Cookbook: Number Understanding of Language Models and How to Improve It
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the NUPA Test, a 41-task benchmark of number understanding and processing across four number formats, evaluates frontier LLMs, and studies tokenizers, positional encodings, data formats, finetuning, and rule-following CoT for improving NUPA.
Problem
LLMs make basic numerical errors (e.g., comparing 9.11 and 9.9) and reasoning benchmarks conflate numerical processing with reasoning, so the basic number understanding and processing ability (NUPA) is poorly measured and improved.
Approach
A systematic benchmark that crosses four number representations (integer, float, fraction, scientific notation) with 17 education-derived tasks to form 41 tasks, with digit-level metrics (digit match, dlength), used to test pretraining tricks, finetuning, and CoT for NUPA.
Key results
- Finetuning Llama-3.1-8B with LoRA on mixed NUPA tasks (about 800 steps) markedly improves NUPA, matching or surpassing GPT-4o on tasks such as max, max-hard, add-float and truediv.
- One-digit tokenizers outperform 2- and 3-digit tokenizers on 0.9B models trained on integer addition, float addition and integer multiplication, in-domain and out-of-domain.
- RF-CoT finetuned Llama-3.1-8B reaches 1.00 exact match on float addition with 5-6 digits and 0.93 on 2-digit fraction multiplication, but is about 17x slower than direct answering and exceeds a 2k context for larger inputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaMA 3.1-8B with RF-CoT rather than directly generating answers | LLaMA 3.1-8B · Add Float, 5 digits | exact match: 0.78 -> 1.00 ±.00 | Helped |
| Fine-tune LLaMA 3.1-8B with RF-CoT rather than directly generating answers | LLaMA 3.1-8B · Multiply Fraction, 2 digits | exact match: 0.50{ $\pm$.02} -> 0.93 ±.01 | Helped |
| Fine-tune LLaMA 3.1-8B with RF-CoT rather than directly generating answers | LLaMA 3.1-8B · Max Scientific, 38 digits | exact match: 0.98{ $\pm$.01} -> 1.00 ±.00; 0.96 -> 1.00 ±.00 | Mixed |
| Fine-tune LLaMA 3.1-8B with RF-CoT rather than directly generating answers | LLaMA 3.1-8B · Mod Integer, 6 digits | exact match: 0.08{ $\pm$.02} -> 0.67 ±.05 | Helped |
| Use RF-CoT instead of direct answer generation | LLaMA 3.1-8B · NUPA Test | sec / sample: 0.371 -> 5.625 | Hurt |
| Use one-digit instead of two- or three-digit tokenization | 0.9B models trained on integer addition, float addition, and integer multiplication · NUPA Test, in-domain and out-of-domain | exact match | Helped |
| Increase tokenizer digit width from one digit to two or three digits | NUPA models · NUPA Test | exact match | Hurt |
| Use random tokenization instead of standard tokenization | number-domain models · Length generalization | exact match | Mixed |
| Use Alibi or NoPE instead of RoPE | 100M models · Integer addition, float addition, fraction multiplication, scientific notation addition | exact match | Helped |
| Use reverse formatting, zero padding, or both instead of vanilla number formatting | models trained on integer and float addition · Length generalization | exact match | Helped |
| Add index hints to number digits | models trained on NUPA tasks · NUPA Test | exact match | No clear effect |
| Finetune LLaMA 3.1-8B on a mixed set of NUPA tasks | LLaMA 3.1-8B · NUPA Test | exact match | Helped |
Surprising findings
- Techniques designed to enhance NUPA (PEs, tokenizers, number formats) are ineffective or harmful when applied to already pretrained models.
- GPT-4o accuracy on the get-digit task in the XL range is only about 20%, barely above random guessing.
Limitations
- The benchmark task range may be incomplete and biased toward certain aspects.
- Only a limited number of models were tested.
- No existing technique was found that fully solves the NUPA problem, and CoT is too costly for complex tasks.
Experimental setup
Datasets: NUPA Test generated training and validation sets
Benchmarks: NUPA Test
Metrics: exact match digit match dlength
Baselines: GPT-4o GPT-4o-mini Llama-3.1-8B Llama-3.1-70B Qwen2-7B Qwen2-72B Mixtral-8x7B Llama-2-7b
Builds on: rule-following chain-of-thought random tokenizer LoRA
Closest papers by meaning
- Efficient numeracy in language models through single-token number embeddings (ICML 2026 · similarity 0.89)
- Interpreting and Improving Large Language Models in Arithmetic Calculation (ICML 2024 · similarity 0.89)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.88)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.88)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.88)
- MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.