Research map / Large language models / Agents, code and math
Evaluating Mathematical Reasoning Beyond Accuracy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReasonEval, LLM-based step-level evaluators scoring validity and redundancy of math reasoning steps, trained on PRM800K, and shows accuracy alone misses reasoning quality issues.
Problem
Final-answer accuracy is the dominant way to evaluate LLM mathematical reasoning, masking logical errors and redundant steps in intermediate reasoning; existing step-level methods rely on references, embeddings, or proprietary prompting.
Approach
Classifies each reasoning step as positive, neutral (valid but no progress), or negative, deriving validity and redundancy scores; implements the evaluator by fine-tuning open LLM backbones (e.g., Llemma, Mistral, WizardMath) with a classification head, trained on PRM800K.
Key results
- ReasonEval-Llemma-34B reaches 79.6 solution-level F1 and 90.8 AUC on MR-MATH-invalid, outperforming GPT-4 (73.2 F1) and Math-shepherd-Mistral-7B (70.1 F1).
- On MR-GSM8K out-of-distribution evaluation, ReasonEval-WizardMath-7B-V1.1 gets 90.7 solution-level AUC and 91.4 step-level AUC on original GSM8K, with step-level results superior to GPT-4.
- Accuracy does not guarantee low false positive rate: WizardMath-7B-V1.1 (31.0% acc) has FPR 16.7% vs WizardMath-70B (18.7% acc) at 16.5%; filtering MMIQC data by validity and redundancy keeps accuracy near full-data SFT with about half the data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a math-specialized Llemma-34B ReasonEval evaluator instead of Math-shepherd-Mistral-7b | Automatic step-reasoning evaluation · MR-MATH-invalid solution-level | F1 Score: 70.1 -> 79.6 | Helped |
| Use ReasonEval WizardMath-7B-V1.1 instead of GPT-4 | Automatic step-reasoning evaluation · MR-MATH-redundant solution-level | F1 Score: 57.1 -> 61.6 | Helped |
| Use Math-shepherd-Mistral-7b instead of ReasonEval Llemma-34B | Automatic step-reasoning evaluation · MR-GSM8K-original solution-level | F1 Score: 86.0 vs. 81.0 | Hurt |
| Increase WizardMath model size from 7B-V1.1 to 70B | WizardMath · MATH | FPR (%): 16.7% vs. 16.5% | No clear effect |
| Filter training data by both validity and redundancy scores instead of random selection | Mistral-7B SFT · MATH | Acc.: 20.0 -> 22.0 | Helped |
| Use validity score p_positive + p_neutral rather than p_positive alone (ablation) | ReasonEval · MR-MATH-invalid solution-level | AUC: 89.7 -> 90.8 | Helped |
| Aggregate validity scores with MIN rather than arithmetic mean (ablation) | ReasonEval · MR-MATH-invalid solution-level | AUC: 89.7 -> 92.8 | Helped |
| Aggregate validity scores with MIN rather than geometric mean (ablation) | ReasonEval · MR-MATH-invalid solution-level | AUC: 90.3 -> 92.8 | Helped |
| Aggregate redundancy scores with arithmetic mean rather than MAX (ablation) | ReasonEval · MR-MATH-redundant solution-level | AUC: 62.7 -> 64.2 | Helped |
| Aggregate redundancy scores with geometric mean rather than MAX (ablation) | ReasonEval · MR-MATH-redundant solution-level | AUC: 62.7 -> 64.2 | Helped |
| Use arithmetic-mean or geometric-mean aggregation rather than MAX for redundancy (ablation) | ReasonEval · MR-MATH-redundant | AUC: AM and GM slightly outperform MAX | Mixed |
Surprising findings
- Increased final-answer accuracy does not inherently reduce the false positive rate of reasoning steps, with WizardMath-7B-V1.1 nearly matching the 70B model's FPR despite much higher accuracy.
- Math-shepherd-Mistral-7B performs best at solution level on MR-GSM8K but lags at step level, attributed to noisy step labels.
Limitations
- Redundancy detection is less accurate and more ambiguous than validity detection, with lower F1 across methods.
- Step-level identification is harder than solution-level, and evaluators depend on step splitting heuristics.
- MR-MATH is small (83 incorrect and 76 correct solutions for validity; 150 each for redundancy), and FPR threshold selection relies partly on a limited human check.
Experimental setup
Datasets: PRM800K MR-MATH MR-GSM8K MATH MMIQC Math-Shepherd
Benchmarks: MR-MATH MR-GSM8K MATH
Metrics: macro F1 AUC false positive rate accuracy
Baselines: ROSCOE-SA ROSCOE-SS GPT-3.5-turbo GPT-4 Math-shepherd-Mistral-7B
Builds on: PRM800K ROSCOE Math-Shepherd MR-GSM8K
Closest papers by meaning
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.91)
- Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals (ICML 2026 · similarity 0.91)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.91)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.91)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.91)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.