Research map / Large language models / Agents, code and math
MIND: Math Informed syNthetic Dialogues for Pretraining LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates zero-shot, math-informed multi-turn synthetic conversations from OpenWebMath web text with LLaMA3-70B-Instruct under seven conversational styles, and shows continued pretraining of a 7B model on them beats raw and rephrased data on math and general reasoning benchmarks.
Problem
Synthetic pretraining data for LLMs, typically rephrasings, adds little complementary knowledge and fails to improve complex multi-hop and mathematical reasoning.
Approach
Prompt an instruction-tuned LLM to convert raw math web text into multi-turn dialogues (with participants holding knowledge gaps) that decompose the content step by step and explain each step in depth, then filter heuristically and mix with raw data for continued pretraining.
Key results
- Continued pretraining on MIND-OWM-4B (Longest Conversation) raises GSM8K from 12.96 to 25.78 and MATH from 4.92 to 6.30 versus raw OWM-4B on a 7B model.
- MIND-OWM-14B (two students) improves GSM8K to 27.29 and MMLU to 49.91 versus 20.47 and 49.49 for OWM-14B.
- Using all conversations from OWM-4B (43B synthetic tokens) outperforms the OWM-14B raw model across GSM8K, MATH, MMLU and general reasoning, with Avg-All 34.80 vs 32.79.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace raw OWM-4B with MIND-generated conversations | 7B LLM continued pretraining · GSM8K | accuracy: 12.96 -> 25.78 | Helped |
| Replace raw OWM-14B with two-student MIND conversations | 7B LLM continued pretraining · GSM8K | accuracy: 20.47 -> 27.29 | Helped |
| Use MIND conversations instead of raw MathPile | 7B LLM continued pretraining · GSM8K | accuracy: 8.79 -> 12.74 | Helped |
| Replace rephrased OWM-4B with MIND conversations | 7B LLM continued pretraining · MATH | accuracy: 5.46 -> 6.30 | Helped |
| Use the two-professors conversation style (ablation) | MIND-OWM-4B · MATH | accuracy: 4.92 -> 4.52 | Hurt |
| Use all seven conversation styles instead of only the longest conversation (ablation) | MIND-OWM-4B · GSM8K | accuracy: 25.78 -> 26.38 | Helped |
| Combine raw OWM-4B with MIND-OWM-4B conversations in a 1:1 ratio (ablation) | 7B LLM continued pretraining · MMLU | accuracy: 45.91 -> 49.57 | Helped |
| Concatenate raw text with all seven generated conversations (ablation) | 7B LLM continued pretraining · General Reasoning (Avg) | accuracy: 52.90 -> 55.10 | Helped |
| Use LLaMA3-8B-Instruct instead of LLaMA3-70B-Instruct to generate conversations (ablation) | MIND-OWM-4B · GSM8K | accuracy: 12.96 -> 22.37 | Helped |
| Train on MIND conversations from OWM-4B rather than the 3.6× larger raw OWM-14B corpus (ablation) | 7B LLM continued pretraining · GSM8K | accuracy: 20.47 -> 26.38 | Helped |
| Select the longest conversation per context rather than use problem-solving conversations (ablation) | MIND-OWM-4B · Avg-All | accuracy: 33.38 -> 34.08 | Helped |
| Generate two-student conversations from MathPile instead of using raw MathPile (ablation) | 7B LLM continued pretraining · General Reasoning (Avg) | accuracy: 54.16 -> 53.98 | No clear effect |
Surprising findings
- The two-professors style, where participants share expertise, yields no gain over raw data, suggesting zero knowledge gap hurts quality.
- LLM-based scoring rated nearly all generations highly, so it was unsuitable as a filter and heuristic length filtering was used instead.
Limitations
- Gains measured on a 7B model with continued pretraining only; full-scale pretraining validated only in appendix with 8B from-scratch runs.
- Synthetic conversations remove special tags, styles and code indentation, so raw data must be mixed in for generality.
- Generated data is distilled from an LLM, so improvements partly reflect distillation; the authors note this.
Experimental setup
Datasets: OpenWebMath MathPile CommonCrawl (FineWebEdu-filtered) pretraining blend (CommonCrawl, books, papers, code, Wikipedia)
Benchmarks: GSM8K MATH MMLU-STEM MMLU ARC-E ARC-C PIQA SIQA HellaSwag WinoGrande OpenBookQA TruthfulQA
Metrics: accuracy average accuracy
Baselines: Raw OpenWebMath Rephrase-OWM-4B OWM-14B MathPile DeepSeekMath corpus
Builds on: OpenWebMath Rephrasing the Web MathPile DeepSeekMath
Closest papers by meaning
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.90)
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.89)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.88)
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.88)
- OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization (NeurIPS 2025 · similarity 0.87)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.