Research map / Large language models / Agents, code and math
GRIP: A Graph-Based Reasoning Instruction Producer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRIP, a knowledge-graph-based pipeline that extracts key concepts from 7.5K MATH seed problems, combines explicit and implicit concept relations to synthesize 2.1M math QA pairs, and filters them with multiple open-source models.
Problem
Existing data synthesis methods for LLM math reasoning scale poorly, produce low-diversity samples that overfit to seed data, and rely on costly closed-source models.
Approach
Build a key-concept relationship graph from seed problems, where explicit edges are co-occurrences and implicit edges are two- or three-hop (hub-based) connections, then synthesize new problems from these unseen concept combinations without including seed data in the prompt, and filter with a multi-open-source-model vote.
Key results
- Mistral-7B trained on GRIP-MATH averages 57.3 across math benchmarks vs 28.0 for base Mistral-7B and 51.2 for prior best MathScale.
- GRIP-trained Qwen2-7B reaches 70.2 average on math benchmarks and 53.4 on MATH, and 57.7 average on scientific reasoning vs 56.0 for base.
- Per-sample synthesis cost is about $0.000057 (0.57 cents in the paper's units), versus 220 for GPT-4-based methods; 71.8% of concept combinations are novel relative to the seed set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Qwen2-7B on GRIP-MATH | Qwen2-7B · Math reasoning benchmarks | AVG: 66.5 -> 70.2 | Helped |
| Train Mistral-7B on GRIP-MATH | Mistral-7B · Math reasoning benchmarks | AVG: 28.0 -> 57.3 | Helped |
| Train Qwen2-7B on GRIP-MATH | Qwen2-7B · Scientific reasoning benchmarks | AVG: 56.0 -> 57.7 | Helped |
| Use GRIP-MATH instead of MathScale data | Mistral-7B · Math reasoning benchmarks | AVG: 51.2 -> 57.3 | Helped |
| Add key concept filtering (ablation) | GRIP question synthesis · Generated questions | AVG Score: 0.61 -> 0.83 | Helped |
| Use hub concepts in three-hop combinations (ablation) | GRIP question synthesis · Three-hop generated questions | AVG Score: 0.39 -> 0.62 | Helped |
| Use more distant concept pairs rather than one-hop pairs (ablation) | GRIP question synthesis · Generated questions | AVG Score: 0.94 -> 0.72; 0.72 -> 0.62 | Hurt |
| Use one-hop, two-hop, and three-hop data instead of duplicated one-hop data (ablation) | Mistral-7B trained on GRIP data · Math | score: 29.3 -> 32.6 | Helped |
| Use one-hop, two-hop, and three-hop data instead of duplicated one-hop data (ablation) | Mistral-7B trained on GRIP data · GSM8K | score: 67.6 -> 71.2 | Helped |
| Add DeepseekR1-7B and DeepseekMath-RL to Qwen2.5-Math-Instruct-7B evaluation (ablation) | GRIP multi-model evaluation · 500 annotated synthesized problems | ACC: 81.4 -> 90.5 -> 95.7 | Helped |
| Use three open-source evaluators instead of GPT-4.1 alone (ablation) | GRIP data evaluation · 500 annotated synthesized problems | ACC: 94.3 -> 95.7 | Helped |
| Add LLM filtering to embedding-based key concept deduplication (ablation) | GRIP concept filtering · Synthesized problems | quality proportion: 26% -> 45% | Helped |
Surprising findings
- Multi-model open-source judging (95.7% accuracy) outperformed GPT-4.1 alone (94.3%) against 500 human labels.
- Training on implicit two-/three-hop data outperformed training on explicit one-hop data (GSM8K 71.2 vs 67.6; MATH 32.6 vs 29.3).
Limitations
- Validated only on mathematical reasoning; effectiveness in other domains is unverified.
- Cross-domain gains are on scientific benchmarks only, with no non-math data used in training.
- Cost comparison relies on assumed token counts and GPU rental pricing rather than measured API spend.
Experimental setup
Datasets: MATH training set GRIP-MATH
Benchmarks: MATH GSM8K GAOKAO-Eval SVAMP ARC-C MMLU-STEM GPQA-Diamond BBH TheoremQA MBPP AIME 2024
Metrics: accuracy average score pass@64 expansion ratio synthesis cost novelty rate similarity score
Baselines: MetaMath WizardMath MathScale MAmmoTH2 MathCoder XwinMath Qwen2-Math DeepSeekMath-RL
Builds on: MetaMath MAmmoTH2 LLaMAFactory vLLM
Closest papers by meaning
- Key-Point-Driven Data Synthesis with Its Enhancement on Mathematical Reasoning (AAAI 2025 · similarity 0.89)
- $\texttt{G1}$: Teaching LLMs to Reason on Graphs with Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.89)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.88)
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.88)
- Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Properties (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.