Research map / Large language models / Agents, code and math
NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NaturalReasoning, a 2.8M-question reasoning dataset backtranslated from web pretraining corpora (DCLM, FineMath) with LLM-generated reference answers, and shows its use in distillation and self-training.
Problem
Scaling LLM reasoning beyond math and coding is limited by a lack of diverse, high-quality, challenging reasoning questions with reference answers.
Approach
Use an LLM to annotate reasoning-rich documents from pretraining corpora (DCLM-baseline, FineMath) and synthesize self-contained challenging questions grounded in them, with reference answers and teacher responses, without human annotation or rule-based extraction.
Key results
- SFT on 1.5M NaturalReasoning examples lets Llama3.1-8B-Base surpass Llama3.1-8B-Instruct on average across MATH, GPQA, MMLU-Pro; OpenMathInstruct-2 and WebInstruct do not surpass it even at 2.8M.
- Distilling DeepSeek-R1 into Llama-3.3-70B-Instruct with 100K NaturalReasoning questions gives average 78.9 vs 79.4 for DeepSeek-R1-Distill-Llama-70B trained on 800K.
- Self-score-filtered DPO on Llama3.1-8B-Instruct reaches 43.67 average on GPQA-Diamond and MMLU-Pro, vs 40.81 baseline and 43.12 with INF-ORM-Llama3.1-70B reward DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill DeepSeek-R1 responses on 100K NaturalReasoning questions instead of s1K-1.1 | Llama-3.3-70B-Instruct · GPQA-Diamond | pass@1: 62.7 -> 63.5 | Helped |
| Increase NaturalReasoning distillation data from 1K to 100K questions | Llama-3.3-70B-Instruct · GPQA-Diamond, MMLU-Pro, and MATH-500 | pass@1: 63.5 -> 67.3; 78.0 -> 79.5; 86.2 -> 89.8 | Helped |
| Use self-score-filtered RFT instead of unfiltered self-scoring (ablation) | Llama3.1-8B-Instruct on SelfTrain-15k · GPQA-Diamond and MMLU-Pro average | accuracy: 42.35 -> 42.54 | Helped |
| Use self-score-filtered DPO instead of unfiltered self-scoring (ablation) | Llama3.1-8B-Instruct on SelfTrain-15k · GPQA-Diamond and MMLU-Pro average | accuracy: 43.22 -> 43.67 | Helped |
| Self-train with reward-scored model responses rather than no additional training | Llama3.1-8B-Instruct on SelfTrain-15k · GPQA-Diamond and MMLU-Pro average | accuracy: 40.81 -> 41.81 (RFT external); 40.81 -> 43.67 (DPO self-score-filtered) | Helped |
| Train with GRPO using NaturalReasoning reference answers and General Verifier | Llama3.1-8B-Instruct · MATH, MMLU-Pro, and GPQA | accuracy: 48.74 -> 52.20; 49.79 -> 49.48; 29.24 -> 31.47 | Mixed |
| Filter SFT examples by agreement with reference answers (ablation) | Llama3.1-8B-Instruct on NaturalReasoning · GPQA-Diamond and MMLU-Pro average | accuracy: 40.87 -> 41.11 | Helped |
| Distill from NaturalReasoning rather than OpenThoughts using 100K examples | Llama3.1-8B-Instruct · GPQA-Diamond, MATH-500, MMLU-Pro, and SuperGPQA | accuracy: 37.8 -> 43.1; 82.2 -> 69.0; 59.0 -> 61.2; 30.0 -> 32.2 | Mixed |
Surprising findings
- Self-rewarding (self-score) methods often matched or outperformed external reward models such as INF-ORM-Llama3.1-70B and Qwen2.5-Math-RM-72B.
- Some datasets such as WebInstruct show performance drops with more data (GPQA 29.02 at 500K to 25.37 at 1.5M), suggesting data quality matters more than volume.
Limitations
- Reference answers are noisy; only 81.68% of questions have answers derived from the pretraining data.
- RL with verifiable rewards experiments are preliminary (50 GRPO steps) and limited by compute.
- Quality and difficulty assessed largely by LLM judges plus a small human study of 100 questions per dataset; potential biases not examined.
Experimental setup
Datasets: DCLM-baseline FineMath GPQA MATH NaturalReasoning OpenThoughts-114k s1K-1.1 LIMO
Benchmarks: MATH MATH-500 GPQA GPQA-Diamond MMLU-Pro SuperGPQA
Metrics: accuracy pass@1 average accuracy high-quality fraction median response length
Baselines: MetaMathQA NuminaMath OpenMathInstruct-2 WebInstruct OpenThoughts-114k s1K-1.1 LIMO DeepSeek-R1-Distill-Llama-70B
Builds on: MetaMath GRPO DPO s1K LIMO
Closest papers by meaning
- Native Reasoning Models: Training Language Models to Reason on Unverifiable Data (ICLR 2026 · similarity 0.87)
- MindGYM: What Matters in Question Synthesis for Thinking-Centric Fine-Tuning? (NeurIPS 2025 · similarity 0.86)
- On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study (NeurIPS 2025 · similarity 0.85)
- Decompose, Analyze and Rethink: Solving Intricate Problems with Human-like Reasoning Cycle (NeurIPS 2024 · similarity 0.85)
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.85)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.