Research map / Large language models / Agents, code and math
RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RealMath, a refreshable benchmark of verifiable research-level math QA pairs auto-extracted from arXiv papers and Math Stack Exchange, and evaluates frontier LLMs on it.
Problem
Existing math benchmarks rely on competition problems, formal proofs, or expert-crafted hard questions, failing to reflect mathematics encountered in actual research; benchmarks also suffer from contamination over time.
Approach
An automated pipeline extracts theorems with single exact answers from research papers and forums, converts them to question-answer pairs with the source context, and filters trivial items, yielding a continually refreshable benchmark with automatic exact-match grading.
Key results
- o3 achieves 49.1% on Math.arXiv and 70.7% on Math.StackExchange; o4-mini reaches 70.8% on Math.StackExchange.
- o3 accuracy drops from 97.5% on easy to 27.9% on hard Math.arXiv questions.
- Omitting context lowers o4-mini on CS.arXiv from 42.3% to 21.6%, showing many theorems are solvable without the paper context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use o3 rather than o4 mini | LLM evaluation on Math.arXiv · Math.arXiv | accuracy: 43.4 -> 49.1 | Helped |
| Use o3 rather than o4 mini | LLM evaluation on CS.arXiv · CS.arXiv | accuracy: 42.3 -> 44.1 | Helped |
| Use o4 mini rather than o3 | LLM evaluation on Math.StackExchange · Math.StackExchange | accuracy: 70.7 -> 70.8 | Helped |
| Increase question difficulty from easy to medium to hard | o3 on Math.arXiv · Math.arXiv difficulty subsets | accuracy: 97.5% -> 81.4% -> 27.9% | Hurt |
| Omit the full theorem context (ablation) | o4-mini on CS.arXiv · CS.arXiv | accuracy: 42.3% -> 21.6% | Hurt |
| Fine-tune on 500 random Math.arXiv samples (ablation) | GPT-4o-mini · Math.arXiv remaining 133 samples | accuracy | No clear effect |
Surprising findings
- Models perform better on 2025 arXiv samples than on 2022 samples, which the authors attribute to training data proximity rather than easier math.
- Fine-tuning GPT-4o-mini on 500 RealMath samples did not improve accuracy.
Limitations
- Source arXiv papers are not peer-reviewed, so some theorems may be incorrect or ambiguous, adding label noise.
- Mathematics Stack Exchange samples are lower quality, requiring heavy filtering.
- Restricted to statements with single exact answers, excluding proofs and inequalities; LLM judge used for filtering and error analysis.
Experimental setup
Datasets: arXiv Mathematics papers (05/2022-09/2022, 12/2024-03/2025) arXiv CS papers (05/2022-10/2023) Math Stack Exchange (04/2024-03/2025)
Benchmarks: RealMath Math.arXiv RealMath CS.arXiv RealMath Math.StackExchange
Metrics: accuracy (exact match)
Baselines: GPT-4o mini Claude 3.5 Sonnet Llama 3.1 405B FrontierMath MATH GSM8K
Builds on: FrontierMath MathConstruct
Closest papers by meaning
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.93)
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.92)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.92)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.92)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.92)
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.