Research map / Large language models / Agents, code and math
Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Putnam-AXIOM, a 522-problem benchmark from Putnam competitions plus a 100-problem functional variation set to test contamination, and proposes Teacher-Forced Accuracy as a step-level proxy metric for reasoning.
Problem
Existing math reasoning benchmarks are saturated and contaminated by training data, and final-answer accuracy alone fails to measure reasoning quality or handle proof-based problems.
Approach
Generate unlimited unseen variants of Putnam problems by programmatically perturbing variables and constants, and score reasoning traces with Teacher-Forced Accuracy, the fraction of reference solution tokens predicted correctly under teacher forcing, as a cheap alternative to PRMs and ROSCOE.
Key results
- o1-preview scores 41.94% (219/522) on Putnam-AXIOM Original, the best among evaluated models, dropping about 19.6 points on Variations (46.8% relative decrease).
- TFA correlates with boxed accuracy on MATH with average absolute correlation 0.662, outperforming all ROSCOE metrics (best ROSCOE average around 0.53).
- LoRA fine-tuning raised accuracy on original problems from 23% to 80% but on variations only from 12% to 33%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate on functional variations instead of the corresponding original questions | o1-preview · Putnam-AXIOM | accuracy: 51 -> 43 | Hurt |
| Evaluate on functional variations instead of the corresponding original questions | GPT-4o · Putnam-AXIOM | accuracy: 30 -> 22 | Hurt |
| Evaluate on functional variations instead of the corresponding original questions | DeepSeek-Distilled-Qwen-32B · Putnam-AXIOM | accuracy: 33 -> 24 | Hurt |
| Evaluate on functional variations instead of the corresponding original questions | Claude-3.5 Sonnet · Putnam-AXIOM | accuracy: 28 -> 27 | Hurt |
| LoRA fine-tune on original problems | the fine-tuned model · 100-question Putnam-AXIOM Original subset | accuracy: 23% -> 80% | Helped |
| LoRA fine-tune on original problems | the fine-tuned model · 100-question Putnam-AXIOM Variation subset | accuracy: 12% -> 33% | Helped |
| Use Teacher-Forced Accuracy rather than the strongest reported ROSCOE metric | proxy metric evaluation · MATH | correlation with boxed accuracy: 0.53 ± 0.05 -> 0.66 ± 0.02 | Helped |
Surprising findings
- Qwen2-Math-7B (base) and other open models score much lower than expected, with most models under 10% on the Original set.
- Removing binary questions causes large accuracy drops for weaker models, indicating guessing inflates their scores.
Limitations
- Only 100 of 522 problems have functional variations, limiting statistical power.
- TFA conditions on a reference solution, so valid alternative reasoning paths may be penalized and tool-use or code-style models under-represented.
- TFA cannot be computed for proprietary models lacking log-probabilities; human correlation of TFA not yet validated.
Experimental setup
Datasets: Putnam competition problems (1938-2023) MATH
Benchmarks: Putnam-AXIOM Original Putnam-AXIOM Variation MATH
Metrics: boxed accuracy Teacher-Forced Accuracy Pearson correlation with boxed accuracy 95% confidence intervals
Baselines: GPT-4o GPT-4 Claude-3.5 Sonnet Qwen2-Math-7B-Instruct DeepSeek-Math-7B-RL NuminaMath-7B ROSCOE metrics BPC
Builds on: MATH ROSCOE LM Harness Functional benchmark variations of MATH
Closest papers by meaning
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.93)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.92)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.91)
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.91)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.91)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.