Research map / Large language models / Agents, code and math
VCR: A “Cone of Experience” Driven Synthetic Data Generation Framework for Mathematical Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VCR, a multi-agent LLM "virtual classroom" that synthesizes SFT math data in three experience stages inspired by Dale's Cone of Experience, with an AdaBoost-style iterative reweighting of hard questions.
Problem
Synthetic SFT data for LLM mathematical reasoning is built by expanding public datasets with a focus on quantity, leading to high cost, low quality density, and homogenization, with challenging samples under-served.
Approach
Simulate a classroom with LLM agents (teacher, TA, students, noter) that first targets questions the model gets wrong (doing), then generates lectures, discussions, and new problem-design "minds" (observation and abstraction), and iterates globally with AdaBoost-style reweighting of hard questions.
Key results
- With 60K data, VCR on LLaMA-2-7B reaches 62.6% GSM8K and 15.2% MATH, versus 58.1% and 12.8% for XwinMathQA.
- On Mistral-7B, VCR reaches 70.5% GSM8K and 27.1% MATH, while SkyworkMathQA reaches 67.6% and 27.8% on MATH.
- Global iterative process (R=3) raises LLaMA-2-7B GSM8K from 62.6% to 66.9% and MATH from 15.2% to 17.8% relative to single-run training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add doing experience through model-specific challenge weighting (ablation) | VCR · GSM8K | accuracy: 60.8 -> 62.6 | Helped |
| add doing experience through model-specific challenge weighting (ablation) | VCR · MATH | accuracy: 13.3 -> 15.2 | Helped |
| add observation experience (ablation) | VCR · GSM8K | accuracy: 61.5 -> 62.6 | Helped |
| add observation experience (ablation) | VCR · MATH | accuracy: 13.9 -> 15.2 | Helped |
| add abstraction experience (ablation) | VCR · GSM8K | accuracy: 62.0 -> 62.6 | Helped |
| add abstraction experience (ablation) | VCR · MATH | accuracy: 14.6 -> 15.2 | Helped |
| add VCR synthetic data instead of XwinMathQA data | LLaMA-2-7B · GSM8K | accuracy: 58.1 -> 62.6 | Helped |
| add VCR synthetic data instead of SkyworkMathQA data | Mistral-7B · MATH | accuracy: 27.8 -> 27.1 | Hurt |
| add abstraction experience (ablation) | VCR · GSM-Distractor | accuracy: 50.0 -> 53.2; 42.2 -> 45.8; 34.8 -> 38.9 | Helped |
Surprising findings
- Removing the problem-design-mind (observation) data lowers performance, suggesting indirect discussion data helps reasoning.
- SkyworkMathQA (reported as 38.1% GSM8K on LLaMA-2-7B) underperforms on GSM8K despite strong MATH results, as it was trained on MATH-only augmentation.
Limitations
- Relies on LLM agents simulating classroom interactions rather than real educational data.
- Reported gains are modest on MATH and the comparison uses differing baseline data sizes and settings.
Experimental setup
Datasets: GSM8K MATH SVAMP ASDiv GSM-Distractor
Benchmarks: GSM8K MATH SVAMP ASDiv GSM-Distractor
Metrics: accuracy
Baselines: Evol-Instruct LEMA MetaMathQA XwinMathQA SkyworkMathQA MuggleMathQA
Builds on: MetaMathQA XwinMathQA SkyworkMathQA LEMA AutoGen
Closest papers by meaning
- RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold (NeurIPS 2024 · similarity 0.88)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.87)
- MIND: Math Informed syNthetic Dialogues for Pretraining LLMs (ICLR 2025 · similarity 0.87)
- Neuro-Symbolic Data Generation for Math Reasoning (NeurIPS 2024 · similarity 0.87)
- MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.87)
- JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.