Research map / Large language models / Reasoning and chain of thought

DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DeepMath-103K, a decontaminated, difficulty-skewed (levels 5-9), answer-verifiable math RL dataset mined from Math StackExchange-style sources, and trains DeepMath models via GRPO that reach SOTA on math benchmarks.

Problem

Existing math RLVR datasets are too easy, contaminated with evaluation benchmarks, lack verifiable answers, and overlap heavily with each other, limiting progress in RL for complex reasoning.

Approach

Source challenging problems from less-structured web forums (Math StackExchange subsets of MMIQC and WebInstructSub), decontaminate semantically against many benchmarks with embedding retrieval plus LLM-judge, filter by GPT-4o-rated difficulty >=5, and keep only problems whose answers agree across three DeepSeek-R1 solutions.

Key results

• DeepMath-Zero-Math-7B reaches 86.9 on MATH-500 and 34.2 on AIME24 starting from Qwen-2.5-Math-7B, versus 46.9 and 11.2 for the base.
• DeepMath-Omn-1.5B reports 64.0 on AIME24 and 57.3 on AIME25, surpassing o1-mini (63.6) on AIME24.
• Source contamination rates before decontamination reached 90% for AIME24 and AMC23 and 76.6% for MATH500.

Surprising findings

• Raw source pools from common open datasets show very high benchmark contamination (e.g., 90% for AIME24 and AMC23).
• SFT on DeepMath R1 solutions still lags behind RL on the same data.

Limitations

• Difficulty labels rely on LLM (GPT-4o) ratings and may be biased.
• Topical diversity is not perfectly balanced.
• Some judgment and multiple-choice questions can be answered by random guessing.

Experimental setup

Datasets: MMIQC WebInstructSub NuminaMath-CoT SimpleRL

Benchmarks: MATH-500 AMC 2023 OlympiadBench Minerva Math AIME 2024 AIME 2025 PolyMath GPQA-Diamond

Metrics: pass@1 averaged over 16 samples

Baselines: Open-Reasoner-Zero-7B Qwen-2.5-Math-7B-SRL-Zoo Qat-Zero-7B Eurus-2-7B-PRIME DeepScaleR-1.5B-Preview Still-3-1.5B-Preview OpenMath-Nemotron-1.5B o1-mini o3-mini (low effort)

Builds on: GRPO Omni-MATH SimpleRL-Zoo verl DeepSeek-R1

Closest papers by meaning

• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model (ICLR 2026 · similarity 0.91)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.89)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.89)
• We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning (ICLR 2026 · similarity 0.89)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.89)