Research map / Large language models / Reasoning and chain of thought
MARGE: Improving Math Reasoning with Guided Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MARGE, which uses a self-generated guide solution and continues sampling from each of its intermediate prefixes to build prefix-shared training data for DPO or RL, improving math reasoning on several 7-8B models without external supervision or value models.
Problem
Self-training for LLM math reasoning plateaus or declines as self-generated data scales, attributed to spurious correlations from insufficient exploration across reasoning stages and poor credit assignment for later steps.
Approach
Hit-guided exploration: pick one correct (for hard queries) or incorrect (for easy queries) self-generated solution as guidance, resample completions from every intermediate prefix of it, estimate state values by Monte Carlo rollouts, and update the guide with on-policy samples each iteration, then train with DPO or a group-relative REINFORCE objective.
Key results
- On Qwen2-7B-Instruct, MARGE-RL reaches 61.08% on MATH pass@1 vs 53.18% base and 59.89% for GRPO, and 68.20% MATH500 pass@64 vs 59.92% base.
- On Llama3.1-8B-Instruct, MARGE-RL improves MATH pass@1 to 54.23% from 49.96% base and OlympiadBench pass@1 to 17.33% from 16.34%.
- On MetaMath-Mistral-7B, MARGE-RL reaches 32.13% MATH pass@1 vs 30.76% for the best baseline (GRPO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MARGE-RL instead of the strongest listed baseline, GRPO | Qwen2-7B-Instruct · MATH | pass@1 accuracy (%): 59.89 -> 61.08 | Helped |
| use MARGE-RL instead of the strongest listed baseline, REINFORCE++ | Qwen2-7B-Instruct · MATH500 | pass@64 accuracy (%): 63.58 -> 68.20 | Helped |
| use MARGE-RL instead of the strongest listed baseline, GRPO | Qwen2-7B-Instruct · GSM8k | pass@1 accuracy (%): 88.07 -> 88.70 | Helped |
| use MARGE-RL instead of the strongest listed baseline, PPO | Qwen2-7B-Instruct · College Math | pass@1 accuracy (%): 35.72 -> 35.77 | Helped |
| use MARGE-RL instead of the strongest listed baseline, PPO | Qwen2-7B-Instruct · College Math | pass@64 accuracy (%): 38.44 -> 40.10 | Helped |
| use MARGE-RL instead of the strongest listed baseline, REINFORCE++ | Qwen2-7B-Instruct · OlympiadBench | pass@1 accuracy (%): 24.49 -> 25.88 | Helped |
| use MARGE-RL instead of the strongest listed baseline, GRPO | Qwen2-7B-Instruct · OlympiadBench | pass@64 accuracy (%): 25.72 -> 27.31 | Helped |
| replace the designed hit-selection heuristic with random guidance selection (ablation) | MARGE-RL · MATH | pass@1 accuracy (%): 61.08 -> 60.21 | Hurt |
| replace the designed hit-selection heuristic with random guidance selection (ablation) | MARGE-RL · GSM8k | pass@1 accuracy (%): 88.70 -> 88.6 | Hurt |
| replace the designed hit-selection heuristic with random guidance selection (ablation) | MARGE-RL · OlympiadBench | pass@1 accuracy (%): 25.88 -> 24.96 | Hurt |
| select a correct response as guidance rather than use the designed hit-selection heuristic (ablation) | MARGE-RL · MATH | pass@1 accuracy (%): 61.08 -> 59.91 | Hurt |
| select a correct response as guidance rather than use the designed hit-selection heuristic (ablation) | MARGE-RL · GSM8k | pass@1 accuracy (%): 88.70 -> 87.59 | Hurt |
Surprising findings
- Adding more self-generated data to SFT, DPO, or REINFORCE baselines often does not help and can hurt, e.g., DPO MATH drops from 57.24 to 51.38 when using the same data as MARGE.
- Running REINFORCE++ on a MARGE-trained Qwen2 model lifts MATH accuracy to 62.34%, above the 59.81% ceiling from the base Instruct model.
Limitations
- Performance gains converge after several iterations, likely due to degrading generation quality.
- Exploration strategy has only been validated on mathematical reasoning.
- Incurs roughly 3-5x more generation computation than vanilla exploration.
Experimental setup
Datasets: MetaMathQA AQuA Omni-Math Big-Math
Benchmarks: MATH MATH500 GSM8k CollegeMath OlympiadBench
Metrics: pass@1 pass@64
Baselines: SFT DPO PPO REINFORCE++ GRPO StepDPO MCTS-DPO ACEMath-7B PRIME-EURUS-7B
Builds on: DPO GRPO REINFORCE++ StepDPO MCTS-DPO
Closest papers by meaning
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.90)
- DenseSteer: Steering Small Language Models towards Dense Math Reasoning (ICML 2026 · similarity 0.90)
- AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking (ICLR 2026 · similarity 0.89)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.89)
- MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy (AAAI 2026 · similarity 0.89)
- CORE: Concept-Oriented Reinforcement for Bridging the Definition–Application Gap in Mathematical Reasoning (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.