Research map / Large language models / Agents, code and math
MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MR-GSM8K, a meta-reasoning benchmark that turns GSM8K-style problems into solution-scoring tasks (correctness, first error step, error reason) and a combined MR-Score, evaluating ~17 open and closed LLMs.
Problem
Standard math benchmarks like GSM8K measure only final-answer accuracy, which ignores reasoning quality and may be inflated by data contamination and overfitting.
Approach
Shift the evaluated model from solver (student) to solution-scorer (teacher): given a question and a candidate step-by-step solution, the model must judge correctness, locate the first error step, and explain the error; these are combined into the MR-Score using MCC and accuracies. Questions include original GSM8K, program-of-thought, and reversed-reasoning variants.
Key results
- GPT-4-Turbo achieves the top MR-Score of 50.5 (zero-shot) and 53.0 (3-shot), well ahead of other models.
- Llama3-70B is the best open-source model at 38.3 zero-shot MR-Score, more than 12 points behind GPT-4-Turbo; Phi3-3.8B (22.9) beats Claude3-Haiku (15.3) and rivals much larger models.
- Specialized math models such as WizardMath-v1.1-7B (0.2) and MetaMath-70B (0.0) collapse on the meta-reasoning task despite strong GSM8K performance; few-shot prompting hurt nearly all models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use three-shot demonstrations instead of zero-shot evaluation | GPT-4-Turbo · MR-GSM8K | MR-Score: 50.5 -> 53.0 | Helped |
| Use three-shot demonstrations instead of zero-shot evaluation | Llama3-70B · MR-GSM8K | MR-Score: 38.3 -> 34.2 | Hurt |
| Use three-shot demonstrations instead of zero-shot evaluation | Deepseek-v2-236B · MR-GSM8K | MR-Score: 29.8 -> 34.1 | Helped |
| Use three-shot demonstrations instead of zero-shot evaluation | Qwen1.5-72B · MR-GSM8K | MR-Score: 20.9 -> 23.3 | Helped |
| Use three-shot demonstrations instead of zero-shot evaluation | Deepseek-Math-7B-RL · MR-GSM8K | MR-Score: 11.6 -> 0.1 | Hurt |
| Use three-shot demonstrations instead of zero-shot evaluation | MAmmoTH-70B · MR-GSM8K | MR-Score: 5.0 -> 0.2 | Hurt |
| Use three-shot demonstrations instead of zero-shot evaluation | general chat models · MR-GSM8K | MR-Score | Mixed |
| Vary the number of correct solutions in three-shot demonstrations (ablation) | evaluated language models · MR-GSM8K | Task 1 TPR and TNR | Mixed |
| Fine-tune Llama-2-70B-base on GSM8K plus GPT-4-generated diagnostic data | Llama2-70B-MR · MR-GSM8K | MR-Score: 10.5% | Helped |
Surprising findings
- Few-shot demonstrations hurt most models, and the label mix of demonstrations shifted true positive and true negative rates, indicating in-context label bias.
- Phi3-3.8B outperforms many models around 20 times its size, suggesting data quality and distillation matter more than scale for this task.
Limitations
- Error-reason evaluation relies partly on GPT-4-Turbo as an automatic judge, which has about 82% accuracy and a higher false positive rate.
- Solutions were generated only by MetaMath-7B and the benchmark is English-only, limiting generality.
- Benchmark is confined to math word problems and does not cover holistic subjects like essay writing.
Experimental setup
Datasets: GSM8K MetaMath-generated solutions
Benchmarks: MR-GSM8K GSM8K
Metrics: MCC TPR TNR first-error-step accuracy error-reason accuracy MR-Score
Baselines: GPT-4-Turbo Claude3-Sonnet Claude3-Haiku GPT-3.5-Turbo Llama3-70B Deepseek-v2-236B Qwen1.5-72B Phi3-3.8B
Builds on: GSM8K chain-of-thought MetaMath ProcessBench-style step-level error identification
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.92)
- MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs (NeurIPS 2024 · similarity 0.91)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.91)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.90)
- Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.