Research map / Large language models / Agents, code and math
UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UGMathBench, a dynamic, multi-version undergraduate-level math benchmark (5,062 problems, 16 subjects, 10 answer types) and proposes EAcc and reasoning gap metrics to evaluate robustness across randomized versions of problems; evaluates 23 LLMs.
Problem
Existing math benchmarks lack broad undergraduate-level coverage and are vulnerable to test-set contamination, and they do not measure whether models solve all variants of a problem consistently.
Approach
Builds a text-only undergraduate benchmark from WebWork programmable problems with three randomized versions per problem, and scores models by EAcc (solving all versions) and the reasoning gap between average per-version accuracy and EAcc.
Key results
- OpenAI-o1-mini achieves the highest EAcc of 56.30% on UGMathBench, with AAcc 68.00% and reasoning gap 11.70%.
- All 23 evaluated LLMs show a reasoning gap; RE ranges from 20.78% (o1-mini) to 196.6% (DeepSeek-MOE-16B-Chat).
- Arithmetic has the highest average EAcc (62.8%), while Abstract Algebra, Differential Equations, and Financial Mathematics average below 10%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OpenAI-o1-mini rather than GPT-4o | LLMs evaluated on UGMathBench · UGMathBench | EAcc: 49.94 -> 56.30 | Helped |
| Use Qwen2-Math-72B-Instruct rather than Qwen2-72B-Instruct | Qwen2 72B models · UGMathBench | EAcc: 35.78 -> 45.85 | Helped |
| Increase Qwen2-Math model size from 7B to 72B | Qwen2-Math-Instruct · UGMathBench | EAcc: 32.46 -> 45.85 | Helped |
| Apply Progressive-Hint Prompting | GPT-4o · Abstract Algebra | EAcc: +7.14% | Helped |
| Apply Progressive-Hint Prompting | GPT-4o · Probability | EAcc: -2.08% | Hurt |
| Apply Progressive-Hint Prompting | Most evaluated LLMs · UGMathBench | EAcc | No clear effect |
| Increase the proportion of one-version test examples used for SFT (ablation) | LLaMA-3-8B · UGMathBench | reasoning gap Δ: 3.43 -> 3.80 -> 4.50 -> 4.64 | Hurt |
| Use the highest contamination proportion reported rather than the lowest (ablation) | LLaMA-3-8B · UGMathBench | reasoning gap Δ: 3.43 -> 4.64 | Hurt |
Surprising findings
- Even the strongest model, OpenAI-o1-mini, solves only 56.3% of problems across all three versions despite ~68% per-version accuracy.
- Errors on the same problem can differ across randomized versions, with about 16.1% of sampled problems showing inconsistent error types.
Limitations
- Text-only; problems requiring images were excluded.
- English-only benchmark.
- Some subjects have few problems, and some problems are judged 'bad questions' (~2.7%).
Experimental setup
Datasets: UGMathBench MetaMathQA
Benchmarks: UGMathBench MATH
Metrics: Acc_v AAcc EAcc reasoning gap robustness efficiency
Baselines: OpenAI-o1-mini GPT-4o GPT-4o-mini Claude-3-Opus Qwen2-Math-72B-Instruct Mistral-Large-Instruct-2407 DeepSeekMath-7B-RL NuminaMath-7B-CoT
Builds on: GSM1K VarBench Progressive-Hint Prompting MetaMathQA
Closest papers by meaning
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.92)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.92)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.91)
- Evaluating Mathematical Reasoning Beyond Accuracy (AAAI 2025 · similarity 0.91)
- UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models (ICML 2025 · similarity 0.91)
- PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.