Research map / Large language models / Agents, code and math
Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds AoPS-Instruct (647K Olympiad-level math QA pairs mined from the AoPS forum with open LLMs) for SFT and LiveAoPSBench, a timestamped, auto-updating contamination-resistant math benchmark, showing performance declines on newer problems.
Problem
Olympiad-level math training data is scarce and expensive to create, and public math benchmarks are prone to contamination, making evaluations unreliable.
Approach
Automatically mine question-answer pairs from the AoPS forum using open LLMs for topic filtering, QA extraction, and step-by-step solution rewriting, and split by timestamp so that the most recent posts form a contamination-resistant, continuously updatable evaluation set.
Key results
- Fine-tuning DeepSeek-Math-7B-Instruct on AoPS-Instruct plus Numina raised LiveAoPSBench-2024 accuracy from 11.7 to 19.7 and OlympiadBench from 14.5 to 25.6.
- Across 18 months of LiveAoPSBench, model accuracy declined over time, with drops ranging from 2.4% to 23.6%.
- Human annotation of 10% of LiveAoPSBench found 92% of annotations marked the automatic answer as correct.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on AoPS-Ins instead of no SFT | DeepSeek-Math 7B-Ins · OlympiadBench | accuracy: 14.5 -> 24.3 | Helped |
| fine-tune on AoPS-Ins instead of no SFT | Mathstral 7B · OlympiadBench | accuracy: 21.20 -> 27.10 | Helped |
| fine-tune on AoPS-Ins instead of no SFT | Llama-3.2 3B-Ins · OlympiadBench | accuracy: 16.1 -> 19.6 | Helped |
| fine-tune on AoPS-Ins instead of no SFT | Llama-3.2 1B-Ins · OlympiadBench | accuracy: 4.70 -> 11.10 | Helped |
| add Numina to AoPS-Ins fine-tuning data (ablation) | DeepSeek-Math 7B-Ins · OlympiadBench | accuracy: 24.3 -> 25.6 | Helped |
| add Numina to AoPS-Ins fine-tuning data (ablation) | Mathstral 7B · MATH | accuracy: 60.80 -> 59.60 | Hurt |
| add Numina to AoPS-Ins fine-tuning data (ablation) | Llama-3.2 3B-Ins · OlympiadBench | accuracy: 19.6 -> 22.8 | Helped |
| add Numina to AoPS-Ins fine-tuning data (ablation) | Llama-3.2 1B-Ins · Omni-Math | accuracy: 11.00 -> 11.70 | Helped |
| rewrite solutions into step-by-step explanations rather than use concise user solutions (ablation) | models fine-tuned on AoPS data · all reported benchmarks | accuracy | Helped |
| use Qwen-2.5 72B rather than Llama-3.1 70B for solution rewriting (ablation) | models fine-tuned on rewritten AoPS data · competition-level math benchmarks | accuracy | Helped |
| use Qwen-2.5 72B rather than Llama-3.1 70B for solution rewriting (ablation) | models fine-tuned on rewritten AoPS data · grade-school math | accuracy | Hurt |
| train on concise, unrevised user solutions instead of rewritten step-by-step solutions (ablation) | models fine-tuned on AoPS data · standard benchmarks | accuracy | Hurt |
Surprising findings
- Solution rewriting by a stronger LLM substantially improves accuracy, and generating full solutions directly from the strong model without user input failed on the example shown.
- Fine-tuning a 7B model on solutions rewritten by a 1.5B model still outperformed both the base model and the rewriting model.
Limitations
- Text-only; omits diagram-dependent problems, especially geometry.
- Evaluation covers only problems with clear final answers; proof-based questions cannot be evaluated.
- Community-generated answers can be noisy; human-annotated answers are not formally verified.
Experimental setup
Datasets: AoPS-Instruct Numina DART-Math-Hard MATH GSM8K OlympiadBench Omni-MATH
Benchmarks: LiveAoPSBench OlympiadBench Omni-MATH MATH AIME24 AMC23
Metrics: accuracy
Baselines: Numina DART-Math-Hard OlympiadBench Omni-MATH NuminaMath-72B-CoT Qwen2.5-Math-72B-Instruct
Builds on: Numina DeepSeekMath decontamination LiveCodeBench
Closest papers by meaning
- Augmenting Math Word Problems via Iterative Question Composing (AAAI 2025 · similarity 0.91)
- MathScale: Scaling Instruction Tuning for Mathematical Reasoning (ICML 2024 · similarity 0.91)
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.90)
- OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization (NeurIPS 2025 · similarity 0.90)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.90)
- Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.