Research map / Large language models / Agents, code and math
OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OccamLLM, which uses a frozen LLM's hidden states to control a symbolic OccamNet that performs exact arithmetic in a single autoregressive step, with a learned switch deciding when to use it.
Problem
LLMs make arithmetic errors, and existing fixes either generate code (slow, insecure) or fine-tune the LLM (risking catastrophic forgetting).
Approach
Decoders map LLM hidden states to weights of a symbolic OccamNet that computes the arithmetic function, and a learned switch chooses between LLM output and OccamNet output per token; the LLM itself stays frozen.
Key results
- OccamLlama 8B and 70B reach 100.0% on all single arithmetic operations (9000 problems), versus 39.5% average for GPT 4o and 99.8% for GPT 4o + Code.
- OccamLlama 70B averages 94.6% on reasoning benchmarks versus 94.2% for GPT 4o and 93.4% for GPT 4o + Code.
- OccamLlama 8B reaches 98.2% on MultiArith Float versus 57.3% for Llama 3 8B and 96.2% for GPT 4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OccamNet-based arithmetic to Llama 3 8B Instruct | OccamLlama 8B · MultiArith Float | accuracy: 57.3 ± 2.0 -> 98.2 ± 0.5 | Helped |
| Add OccamNet-based arithmetic to Llama 3 8B Instruct | OccamLlama 8B · MATH401 | accuracy: 60.3 ± 2.4 -> 85.0 ± 1.8 | Helped |
| Add OccamNet-based arithmetic to Llama 3 70B Instruct | OccamLlama 70B · Reasoning benchmarks average | accuracy: 90.0 ± 1.2 -> 94.6 ± 0.9 | Helped |
| Add OccamNet-based arithmetic to Llama 3 8B Instruct | OccamLlama 8B · GSM8K | accuracy: 79.8 ± 1.1 -> 73.5 ± 1.2 | Hurt |
| Add OccamNet-based arithmetic to Llama 3 8B Instruct | OccamLlama 8B · Single Eq | accuracy: 96.3 ± 0.8 -> 92.9 ± 1.1 | Hurt |
| Add OccamNet-based arithmetic to Llama 3 8B Instruct | OccamLlama 8B · MultiArith | accuracy: 99.8 ± 0.2 -> 99.2 ± 0.4 | No clear effect |
| Add OccamNet-based arithmetic to Llama 3 8B Instruct | OccamLlama 8B · AddSub | accuracy: 93.4 ± 1.2 -> 91.6 ± 1.4 | No clear effect |
| Use OccamLlama instead of GPT 4o | OccamLlama 8B · MultiArith Float | accuracy: 96.2 ± 0.8 -> 98.2 ± 0.5 | Helped |
| Use OccamLlama instead of GPT 4o + Code | OccamLlama 70B · Reasoning benchmarks average | accuracy: 93.4 ± 1.7 -> 94.6 ± 0.9 | No clear effect |
| Use a two-layer Complete OccamNet rather than the standard single-layer OccamNet | OccamLlama 8B · Multistep arithmetic average | accuracy: 58.7 ± 0.9 -> 98.1 ± 0.3 | Helped |
Surprising findings
- OccamLlama 8B underperforms Llama 3 8B on GSM8K and Single Eq because the switch wrongly triggers OccamNet on fractions and percentages.
- GPT 4o + Code Interpreter does not outperform GPT 4o on average across the math benchmarks.
Limitations
- Single-layer OccamNet handles only single two-input operations, so compound expressions and percentages/fractions fail without extra handling.
- Switch trained on limited data, causing incorrect OccamNet triggers on out-of-distribution text such as GSM8K and Single Eq.
- Llama sometimes appends extra digits after OccamNet outputs, requiring a workaround.
Experimental setup
Datasets: MultiArith (training set, modified) synthetic arithmetic prompts
Benchmarks: synthetic arithmetic (+,-,x,/,sqrt,exp,log,sin,cos) AddSub GSM8K MultiArith MultiArith Float MATH401 Single Eq SVAMP
Metrics: accuracy (%) standard error of mean relative error (%)
Baselines: Llama 2 7B Chat Llama 3 8B Instruct GPT 3.5 Turbo GPT 4o GPT 4o + Code Interpreter Llama 3 70B Instruct
Builds on: OccamNet Toolformer REINFORCE
Closest papers by meaning
- Interpreting and Improving Large Language Models in Arithmetic Calculation (ICML 2024 · similarity 0.88)
- Efficient numeracy in language models through single-token number embeddings (ICML 2026 · similarity 0.88)
- Benchmarking Large Language Models with Integer Sequence Generation Tasks (NeurIPS 2025 · similarity 0.87)
- Llemma: An Open Language Model for Mathematics (ICLR 2024 · similarity 0.87)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.86)
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.