Research map / Large language models / Reasoning and chain of thought
Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that pretraining GPT2-style models on synthetic iGSM math data containing injected errors followed by immediate corrections ("retry data") improves reasoning accuracy over error-free pretraining at equal token budget, while retry-upon-regret inference and LoRA finetuning provide much smaller gains.
Problem
Whether language models can learn to correct their own reasoning mistakes immediately during generation, and whether including error-correction data in pretraining (rather than relying on multi-round self-correction prompting) improves reasoning accuracy over training on error-free data.
Approach
Uses fully controllable synthetic iGSM math data to insert erroneous steps followed by a [BACK] token and correct steps into pretraining data, showing that this retry data (without label masking) raises accuracy versus error-free data at the same token budget, and that error-correction is a skill not easily acquired by LoRA finetuning or by probing-guided retries.
Key results
- Pretraining with retry data (retry_rate 0.5) raises accuracy on iGSM-med qp op=23 from 78% to 94% versus error-free data.
- Retry upon regret with probing (version1/2, 99% detection) improves accuracy only ~2% over beam32 (e.g., 78% to 80% on iGSM-med pq op=23).
- Models pretrained on retry data with retry_rate=0.2 retry less than 0.3 times on average, and still produce shortest solutions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on retry data rather than the same-token amount of error-free data | GPT2-12-12 · iGSM-med qp op=23 | accuracy: 78% to 94% | Helped |
| Pretrain on retry data rather than error-free data | GPT2-12-12 · iGSM-med pq op=23 | accuracy: 78% to 95% | Helped |
| Use retry upon regret instead of original generation | GPT2-12-12 · iGSM-med pq op=23 | accuracy: 78% to 80% | Helped |
| Increase retry_rate within a reasonable range (ablation) | GPT2-12-12 pretrained on retry data · iGSM reasoning tasks | accuracy | Helped |
| Add label masking on erroneous tokens (ablation) | GPT2-12-12 pretrained on retry data · iGSM reasoning tasks | accuracy | No clear effect |
| Fine-tune an error-free pretrained model with retry data using LoRA (ablation) | GPT2-12-12 · iGSM reasoning tasks | accuracy | No clear effect |
| Use small-rank LoRA fine-tuning on retry data instead of error-free pretraining (ablation) | GPT2-12-12 · iGSM reasoning tasks | accuracy | Hurt |
| Use full fine-tuning on retry data instead of LoRA fine-tuning (ablation) | GPT2-12-12 pretrained on error-free data · iGSM reasoning tasks | accuracy | Helped |
| Use retry_weak fake mistakes instead of perfectly designed retry data (ablation) | GPT2-12-12 · iGSM reasoning tasks | accuracy | Helped |
| Use retry_miss fake mistakes instead of perfectly designed retry data (ablation) | GPT2-12-12 · iGSM reasoning tasks | accuracy | Helped |
Surprising findings
- Label masking of erroneous tokens is unnecessary; vanilla autoregressive training on retry data works well.
- LoRA finetuning on retry data from an error-free pretrained model fails to match pretraining on retry data, even underperforming error-free pretraining at small ranks.
Limitations
- Experiments are on synthetic iGSM data with GPT2-small, not real-world math or large LLMs.
- Retry-upon-regret gains depend on a near-perfect error detector and increase inference cost.
- Fake-mistake (retry_weak/retry_miss) variants are only tested in synthetic settings.
Experimental setup
Datasets: iGSM-med iGSM-hard GSM8K (referenced for design)
Benchmarks: iGSM-med (OOD op 20-23, reask) iGSM-hard (OOD op 28-32)
Metrics: accuracy (fully-correct intermediate steps) average retries per problem unnecessary operations per solution
Baselines: beam search error-free pretraining retry upon regret (version1/version2/versionP) LoRA finetuning on retry data full finetuning (continued pretraining)
Builds on: iGSM synthetic math dataset Physics of Language Models Part 2.1 LoRA GPT-2
Closest papers by meaning
- Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process (ICLR 2025 · similarity 0.93)
- MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.91)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.91)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.91)
- RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold (NeurIPS 2024 · similarity 0.90)
- MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.