Research map / Large language models / Agents, code and math
HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HARDMath, an algorithmically generated benchmark of graduate-level applied math problems (asymptotic/dominant-balance methods) with numerically validated solutions, and shows leading LLMs score low (best 62.3% for o1-mini 5-shot CoT on HARDMath-mini).
Problem
Advanced applied mathematics problems requiring analytical approximation techniques (asymptotic methods, dominant balance) are underrepresented in LLM benchmarks, and current LLMs struggle with them.
Approach
Auto-generate graduate-level applied math problems (nondimensionalization, polynomial root approximation, nonlinear ODE asymptotics, traditional and Laplace integrals) with SymPy/SciPy solutions validated against numerical ground truth, enabling arbitrarily large datasets; evaluate with boxed-answer matching plus GPT-4o procedural grading.
Key results
- o1-mini with 5-shot CoT achieves the highest overall accuracy of 62.3% on HARDMath-mini, while GPT-4 5-shot CoT reaches 43.8%.
- Best open-source model Llama3-8b with 5-shot CoT reaches 20.2% overall accuracy.
- GPT-4 scores 28.1% on 40 hand-crafted word problems without problem-specific hints.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 5-shot CoT instead of 0-shot prompting | GPT-3.5 · HARDMath-mini | accuracy (ALL): 6.04 -> 24.6 | Helped |
| Use 5-shot CoT instead of 0-shot prompting | GPT-4 · HARDMath-mini | accuracy (ALL): 14.0 -> 43.8 | Helped |
| Use 5-shot CoT instead of 0-shot CoT | o1-mini · HARDMath-mini | accuracy (ALL): 29.8 -> 62.3 | Helped |
| Use 5-shot CoT instead of 0-shot prompting | Llama3-8b · HARDMath-mini | accuracy (ALL): 3.67 -> 20.2 | Helped |
| Use 5-shot CoT instead of 0-shot prompting | CodeLlama-13b · HARDMath-mini | accuracy (ALL): 1.94 -> 9.79 | Helped |
| Evaluate GPT-4 on applied-science contextual word problems without problem-specific hints | GPT-4 · 40 applied-science word problems | accuracy: 28.1% | Hurt |
Surprising findings
- o1-mini, despite official guidance recommending simple prompting, improves substantially with 5-shot CoT over 0-shot.
- Frontier models score far below their results on MATH and GSM8K, indicating the benchmark is much harder.
Limitations
- Validation relies on <10% numerical error thresholds and semi-automated visual human checks rather than full manual verification.
- Evaluation models limited to a few closed and open models; o1-mini intermediate steps often not shown.
- Word-problem set is small (40 problems) and context generation is preliminary.
Experimental setup
Datasets: HARDMath
Benchmarks: HARDMath-mini HARDMath word problems (40)
Metrics: accuracy partial credit score
Baselines: GPT-3.5 GPT-4 o1-mini Llama3-8b CodeLlama-13b
Builds on: chain-of-thought prompting MATH GSM8K
Closest papers by meaning
- HARDMath2: A Benchmark for Applied Mathematics Built by Students as Part of a Graduate Class (NeurIPS 2025 · similarity 0.95)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.89)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.89)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.89)
- OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization (NeurIPS 2025 · similarity 0.89)
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.