Research map / Large language models / Agents, code and math
Augmenting Math Word Problems via Iterative Question Composing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the MMIQC dataset (web math data plus GPT-generated question-response pairs, including the IQC iterative question-composing augmentation) and fine-tunes open base LLMs on it to reach new open-source SOTA on MATH.
Problem
Improving competition-level math word problem solving of open-source LLMs without external tools, where a large gap remains to closed models such as GPT-4.
Approach
Iterative Question Composing (IQC) repeatedly asks a strong LLM (GPT-4) to compose harder multi-step questions from seed problems, with rejection sampling by GPT-3.5 to filter answers, and combines this synthetic data with reused web math corpora (MathStackExchange from RedPajama) for SFT.
Key results
- Qwen-72B-MMIQC reaches 45.0% on MATH zero-shot, exceeding the previous open-source SOTA (36.8%) by 8.2 points and GPT-4 (2023-0314) at 42.5%.
- Mistral-7B-MMIQC reaches 36.0% on MATH vs 28.2% for Mistral-7B-MetaMathQA; Llemma-34B-MMIQC 38.6% vs 34.8%; DeepSeek-67B-MMIQC 41.0% vs 36.8%.
- Ablation on Mistral-7B: IQC adds about 3.1 points (31.5% to 34.4% over iterations 1-4 cumulative, with iteration 1 alone reaching 33.0%) using only 55.1K samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Mistral-7B on MMIQC instead of MetaMathQA | Mistral-7B · MATH | accuracy: 28.2 -> 36.0 | Helped |
| Fine-tune Llemma-34B on MMIQC instead of MetaMathQA | Llemma-34B · MATH | accuracy: 34.8 -> 38.6 | Helped |
| Fine-tune DeepSeek-67B on MMIQC instead of MetaMathQA | DeepSeek-67B · MATH | accuracy: 36.8 -> 41.0 | Helped |
| Fine-tune Qwen-72B on MMIQC instead of MetaMathQA | Qwen-72B · MATH | accuracy: 41.7 -> 45.0 | Helped |
| Use the smaller MetaMathQA subset instead of the full dataset (ablation) | Mistral-7B fine-tuned on MetaMathQA · MATH | accuracy: {-1.8} | Hurt |
| Add AnsAug & QB (ablation) | Mistral-7B trained on the MetaMathQA subset · MATH | accuracy: {+1.9} | Helped |
| Add AugSimilar (ablation) | Mistral-7B trained on MetaMathQA subset + AnsAug & QB · MATH | accuracy: {+3.3} | Helped |
| Add IQC iteration 1 (ablation) | Mistral-7B trained on prior MMIQC subsets · MATH | accuracy: {+4.8} | Helped |
| Add IQC iteration 2 (ablation) | Mistral-7B trained with IQC iteration 1 and prior subsets · MATH | accuracy: {+5.5} | Helped |
| Add IQC iterations 3 & 4 (ablation) | Mistral-7B trained with earlier IQC iterations and prior subsets · MATH | accuracy: {+6.2} | Helped |
| Add MathStackExchange (ablation) | Mistral-7B trained on the other MMIQC subsets · MATH | accuracy: {+7.8} | Helped |
| Set maximal learning rate to 1e-6 instead of 1e-5 (ablation) | Mistral-7B fine-tuned on MMIQC · MATH | accuracy: 36.0 -> 32.3 | Hurt |
Surprising findings
- A filtered 204K subset of MetaMathQA loses only 1.8 points versus the full 395K set, suggesting redundancy in the original k=20 sampling strategy.
- Only 2 of 100 checked IQC QA pairs were incorrect (85% correct), and some composed questions are not strict sub-problems of their predecessors.
Limitations
- Synthetic data depends on GPT-4 and GPT-3.5 outputs, so generation relies on closed proprietary APIs.
- Results are zero-shot pass@1 without tools or verifiers and cannot be directly compared with sampling/verifier-based methods.
- Some IQC-composed questions do not strictly follow the sub-problem requirement; 85% correctness on a 100-sample check.
Experimental setup
Datasets: MATH GSM8K MetaMathQA RedPajama OpenWebMath
Benchmarks: MATH 2023 Hungarian National High School Mathematics Finals
Metrics: pass@1 accuracy exam grade out of 117
Baselines: MetaMath-7B Mistral-7B-MetaMathQA Llemma-34B-MetaMathQA DeepSeek-67B-MetaMathQA Qwen-72B-MetaMathQA GPT-4 Gemini-Ultra Minerva-540B ToRA-Code-7B MAmmoTH-Coder-7B
Builds on: MetaMathQA MetaMath question bootstrapping Llemma RedPajama DeepSpeed ZeRO
Closest papers by meaning
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.91)
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (ICLR 2024 · similarity 0.89)
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.89)
- MathScale: Scaling Instruction Tuning for Mathematical Reasoning (ICML 2024 · similarity 0.89)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.88)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.