Research map / Large language models / Agents, code and math
HARDMath2: A Benchmark for Applied Mathematics Built by Students as Part of a Graduate Class
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HARDMath2, a 211-problem graduate applied-math benchmark (asymptotics, WKB, boundary layers, nonlinear PDEs, integrals) written and verified by students in a Harvard course, with automatic SymPy-based parsing of final boxed answers, and evaluates frontier LLMs.
Problem
Existing math benchmarks emphasize exact-answer or proof problems and rely on LLM or manual grading, leaving approximation-based applied mathematics (asymptotics, WKB, boundary layers) under-covered.
Approach
Students in a graduate applied math course author and peer-verify original approximation problems, using a Google Sheets-integrated evaluator that parses LaTeX boxed answers into SymPy and checks numerical equivalence at random variable values, and iteratively make problems harder by probing LLM failure modes.
Key results
- Gemini 2.5 Flash Thinking achieves the highest overall pass@1 at 60.1%, with Gemini 2.5 Pro Preview at 57.7%.
- o3 scores 52.5% overall, followed by o4-mini at 46.1% and o3-mini at 46.0%.
- Nonlinear PDE problems are hardest: only top models exceed 35%, e.g. o3-mini 37.8% and Gemini 2.5 variants 44.3-45.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gemini 2.5 Flash Thinking instead of Gemini 2.0 Flash | Gemini models · HARDMath2 overall | Pass@1: 8.5 -> 60.1 | Helped |
Surprising findings
- Gemini 2.0 Flash scores only 8.5% overall, far below the 2.5 Flash Thinking model's 60.1%.
- Several models score 0% on asymptotic series problems (e.g., DeepSeek V3, GPT-4.1, o3-mini), while integrals are easiest with many models above 70%.
Limitations
- Evaluation uses only a single pass@1 per model and a limited set of models accessed via APIs.
- Parser cannot handle all valid answer forms and requires problems to be phrased to yield unique parsable answers.
- Dataset is small (211 problems) with uneven per-category sizes.
Experimental setup
Datasets: HARDMath2
Benchmarks: HARDMath2
Metrics: pass@1
Baselines: Claude 3.7 Sonnet DeepSeek V3 Llama 4 Maverick GPT-4.1 GPT-4o o1 o3 o3-mini o4-mini Gemini 2.0 Flash Gemini 2.5 Flash Thinking Gemini 2.5 Pro Preview
Builds on: HARDMath SymPy
Closest papers by meaning
- HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics (ICLR 2025 · similarity 0.95)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.89)
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (ICLR 2024 · similarity 0.88)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.88)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.87)
- Llemma: An Open Language Model for Mathematics (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.