Research map / Large language models / Reasoning and chain of thought
Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that under a fixed sampling FLOPs budget, synthetic reasoning data from a weaker-but-cheaper LM (Gemma2-9B, Gemini-1.5-Flash) yields better finetuned reasoners than data from a stronger-but-expensive LM, across distillation, self-improvement, and a new weak-to-strong setup.
Problem
Whether synthetic reasoning data from strong but expensive LMs is compute-optimal for training LM reasoners under a fixed inference/sampling budget, versus sampling more from weaker cheaper LMs.
Approach
Compute-matched sampling: since one SE sample costs P_SE/P_WC WC samples, compare datasets at equal FLOPs; WC data yields higher coverage and diversity (despite higher false positive rate), and finetuning on it beats SE data across distillation, self-improvement, and weak-to-strong improvement setups.
Key results
- Gemma2-9B (WC) gives 11% higher coverage and 86% higher diversity than Gemma2-27B (SE) on MATH at low budget, with ~7% higher FPR.
- Finetuning Gemma-7B on Flash (WC) vs Pro (SE) data at price-matched budget gives 31.6% relative gain on MATH.
- Weak-to-strong improvement: finetuning Gemma2-27B on Gemma2-9B data beats Gemma2-27B self-generated data by 5.8% (low budget) and 4.3% (high budget) relative on MATH.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace compute-matched Gemma2-27B synthetic data with Gemma2-9B data | Gemma-7B knowledge distillation · MATH and GSM-8K, low and high budgets | pass@1 accuracy: MATH: 6% (low), 5.8% (high); GSM-8K: 4.2% (low), 1.3% (high) | Helped |
| Train on self-generated Gemma2-9B data instead of Gemma2-27B data | Gemma2-9B self-improvement · MATH, low and high budgets | pass@1 accuracy: 3.8% (low); 2% (high) | Helped |
| Train on self-generated Gemma2-9B data instead of Gemma2-27B data | Gemma2-9B self-improvement · GSM-8K, high budget | pass@1 accuracy | No clear effect |
| Train Gemma2-27B on Gemma2-9B data instead of its own data | Gemma2-27B weak-to-strong improvement · MATH and GSM-8K, low and high budgets | pass@1 accuracy: MATH: 5.8% (low), 4.3% (high); GSM-8K: 1.2% (low), 1.5% (high) | Helped |
| Use compute-matched Gemma2-9B sampling instead of Gemma2-27B sampling | Synthetic data generation · MATH | coverage: 11% (low budget); 6% (high budget) | Helped |
| Use compute-matched Gemma2-9B sampling instead of Gemma2-27B sampling | Synthetic data generation · MATH and GSM-8K | diversity: MATH: 86% (low), 125% (high); GSM-8K: 134% (low), 158% (high) | Helped |
| Use compute-matched Gemma2-9B sampling instead of Gemma2-27B sampling | Synthetic data generation · MATH and GSM-8K | false positive rate: 7% higher on MATH; 2% higher on GSM-8K | Hurt |
| Replace Gemini-1.5-Pro samples with price-matched Gemini-1.5-Flash samples | Gemma-7B, Gemma2-9B, and Gemma2-27B knowledge distillation · MATH | pass@1 accuracy: 31.6% (Gemma-7B); 14.4% (Gemma2-9B); 10.9% (Gemma2-27B) | Helped |
| Reduce Flash sampling to 5 solutions per problem, at 0.15× Pro cost (ablation) | Gemma-7B, Gemma2-9B, and Gemma2-27B · MATH | pass@1 accuracy: 19.1% (Gemma-7B); 9.8% (Gemma2-9B); 5.7% (Gemma2-27B) | Helped |
| Use Flash-generated instruction responses instead of Pro-generated responses | Gemma2-9B instruction following · IFEval | instruction-level accuracy: 12.8% | Helped |
| Reduce the MATH training set to 500 problems while retaining compute-matched sampling (ablation) | Gemma-7B, Gemma2-9B, and Gemma2-27B finetuning · MATH | accuracy of WC data over SE data: 12.93% (student); 11.4% (WC-LM); 5.1% (SE-LM) | Helped |
| Keep diverse correct solutions rather than one correct solution per question (ablation) | Three finetuning setups · MATH | finetuned model performance: diversity 11 to 1 | Helped |
Surprising findings
- Number-matched WC data (one sample per problem) underperforms SE data; gains appear only under compute-matched sampling.
- Without ground-truth filtering or with LM-judge filtering, SE data slightly outperforms WC data for Gemma models, so the WC advantage is context-dependent.
Limitations
- Relies on final-answer correctness filtering with gold labels for main results; FPR of WC data is higher.
- Compute matching uses parameter ratio and assumes equal tokens per solution and ignores hardware; for Gemini, price used as proxy for compute.
- Coding results are context-dependent; the benefit of WC data is not uniform across tasks.
Experimental setup
Datasets: MATH GSM-8K MBPP OpenAssistant1 Functional MATH
Benchmarks: MATH GSM-8K Functional MATH HumanEval IFEval
Metrics: pass@1 accuracy maj@k coverage@k diversity@k false positive rate instruction-level accuracy
Baselines: human-written solutions Gemma2-27B-generated data (SE data) number-matched WC data
Builds on: STaR pass@k coverage estimation knowledge distillation self-improvement
Closest papers by meaning
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.91)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.91)
- Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus (NeurIPS 2024 · similarity 0.91)
- ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation (AAAI 2026 · similarity 0.90)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.90)
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.