Research map / Large language models / Agents, code and math
Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Omni-MATH, a 4428-problem text-only Olympiad-level math benchmark with domain and difficulty labels, plus the open Omni-Judge answer verifier, and evaluates 15 LLMs.
Problem
Existing math benchmarks such as GSM8K and MATH are saturated, and few Olympiad-level benchmarks focus exclusively on text-only mathematics with fine-grained domain and difficulty labels; answer evaluation for diverse Olympiad answer formats is also hard with rule-based checking.
Approach
Builds a text-only Olympiad math benchmark from contest pages and AoPS sources, annotated by humans, with a hierarchical 33+ sub-domain tree and AoPS-based difficulty ratings (0-10), and uses a GPT-4o or fine-tuned open judge (Omni-Judge) for answer verification instead of fixed rules.
Key results
- OpenAI o1-mini reaches 60.54% accuracy on Omni-MATH, the best overall, while o1-preview reaches 52.55%.
- The best vanilla open-source model, Qwen2.5-MATH-72b-Instruct, reaches 36.20% accuracy.
- GPT-4o agrees with human annotations 98% of the time on a 100-sample meta-evaluation; Omni-Judge (LLaMA-3.1-8B based) reaches 86% human agreement and over 91% consistency with GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase Best-of-N reward-model selection from RM@8 to RM@256 | Qwen2.5-MATH-72b-Instruct · Omni-MATH | accuracy: 36.34 -> 35.95 | Hurt |
| Increase Best-of-N reward-model selection from RM@8 to RM@256 | Qwen2.5-MATH-7b-Instruct · Omni-MATH | accuracy: 35.70 -> 35.79 | Helped |
| Use Best-of-N test-time scaling | Qwen2.5-MATH models · Omni-MATH | accuracy: 35.70; 35.79; 36.34; 35.95 | Mixed |
| Use more difficult in-context examples | GPT-4o · Omni-MATH | accuracy | Helped |
| Use T4 rather than less difficult in-context examples | GPT-4o · Omni-MATH | accuracy | Hurt |
| Increase domain similarity of in-context examples | GPT-4o · Omni-MATH | accuracy | Helped |
Surprising findings
- Best-of-N scaling with Qwen2.5-Math-RM-72B did not consistently improve results; RM@256 did not beat RM@8 for the 72B model.
- Models perform worse on discrete mathematics than on algebra and calculus.
Limitations
- Judge reliability is measured on only 100 human-annotated samples.
- Reasoning models were limited to a 4096-token completion budget, which may understate their accuracy.
- Benchmark leakage analysis relies on an n-gram proxy that finds little contamination.
Experimental setup
Datasets: Omni-MATH AoPS Wiki AoPS Forum Contest pages (IMO, IMC, HMMT, etc.)
Benchmarks: Omni-MATH GSM8K MATH
Metrics: accuracy consistency with GPT-4o agreement with human annotation
Baselines: GPT-4o Qwen2.5-MATH-72b-Instruct NuminaMATH-72B-COT Claude-3.5-Sonnet MetaLlama-3.1-70B-instruct DeepSeek-Coder-V2 Mathstral-7B-v0.1 Qwen2-MATH-72b-Instruct
Builds on: MATH AoPS Wiki difficulty ratings GPT-4o judge
Closest papers by meaning
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.90)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.90)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.90)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.89)
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.89)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.