Research map / Large language models / Agents, code and math
MathArena: Evaluating LLMs on Uncontaminated Math Competitions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MathArena, a contamination-free benchmark that evaluates LLMs on freshly released 2025 math competitions (final-answer and human-graded proof problems) and publishes a live leaderboard.
Problem
Existing math benchmarks such as AIME 2024 are public and likely contaminated, private benchmarks are costly and non-reproducible, and most benchmarks ignore proof-writing ability.
Approach
Evaluate models immediately after new recurring competitions (AIME, HMMT, BRUMO, CMIMC, USAMO, IMO, Project Euler) are released, so models released before the competition are uncontaminated; combine rule-based sympy parsing with an LLM judge for answers and expert human grading for proofs.
Key results
- GPT-5 achieved about 91.3% average accuracy across the four 2025 numerical competitions, with Grok 4 at 90.6% and Grok 4 Fast at 90.4%.
- On IMO 2025, GPT-5 scored 16/42 (about 38%), below the 19-point bronze threshold.
- AIME 2024 shows substantial contamination: most models score 10-20 points above the human-aligned expectation, while HMMT shows much smaller discrepancy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate models on newly released competitions rather than older public competitions | LLM evaluation on math competitions · AIME 2024 versus AIME 2025 | accuracy relative to human performance: 10%-20%; nearly 60% | Mixed |
| Use chain-of-thought reasoning models instead of pre-September-2024 models | LLMs evaluated on HMMT 2025 · HMMT 2025 | accuracy: less than $10\%$ | Helped |
| Use best-of-n selection over 32 generated proofs | IMO 2025 proof evaluation · IMO 2025 | total score: below $10\%$; $38\%$ | Helped |
| Solve proof-based problems instead of numerical-answer problems | GPT-5 · IMO 2025 versus numerical competitions | score: 91.25; $38\%$ | Hurt |
| Evaluate on combinatorics and geometry problems rather than algebra and number theory | LLMs · MathArena 2025 numerical competitions | accuracy: 48.8; 55.5; 64.5; 68.6 | Mixed |
| Use Grok 4 Fast instead of Grok 4 | Grok 4 · All numerical competitions | average cost: 7.556594; 0.184516 | Helped |
Surprising findings
- QwQ-Preview-32B outperforms the expected human-aligned AIME 2024 performance by nearly 60%, indicating extreme contamination.
- Some unselected proof responses were more coherent than the best-of-n selected ones, though they contained more factual errors.
Limitations
- Small number of competition problems (162) yields wide confidence intervals.
- Answer-based competitions may saturate soon, requiring harder competitions.
- Residual contamination risk from time gap between competition release and model evaluation for closed models.
Experimental setup
Datasets: AIME 2025 HMMT 2025 BRUMO 2025 CMIMC 2025 USAMO 2025 IMO 2025 Project Euler
Benchmarks: AIME 2025 HMMT 2025 BRUMO 2025 CMIMC 2025 USAMO 2025 IMO 2025 Project Euler
Metrics: pass@1 accuracy averaged over 4 runs score out of 42 for proofs rank confidence intervals cost (USD)
Baselines: GPT-5 Grok 4 Gemini-2.5-Pro o3 o4-mini DeepSeek-R1-0528 DeepSeek-v3.2 GPT-OSS-120B Claude-Sonnet-4.5 QwQ-32B-Preview Qwen3-235B-A22B
Builds on: GSM8K MATH FrontierMath Humanity's Last Exam LiveBench
Closest papers by meaning
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.92)
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.92)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.91)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.91)
- MathConstruct: Challenging LLM Reasoning with Constructive Proofs (ICML 2025 · similarity 0.91)
- One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.